技术领域Technical Field
本申请涉及图像处理领域，尤其涉及一种图像处理方法及电子设备。The present application relates to the field of image processing, and in particular to an image processing method and electronic equipment.
背景技术Background technique
随着计算机视觉(computer vision，CV)技术的快速发展和芯片算力的大幅提升，图像处理技术的得到普遍应用，极大地提升了图像质量。With the rapid development of computer vision (CV) technology and the substantial improvement of chip computing power, image processing technology has been widely used, greatly improving image quality.
目前大部分应用于人物图像的图像处理技术，使用由大量非特定人像的照片训练得到的图像处理模型对当前人物图像进行增强处理，处理后的人物图像中人物的特点可能会包含不属于当前人物的特点，例如，增加了并不属于当前人像的深邃的眼窝、高挺的鼻子山根等。Currently, most image processing technologies applied to human images use image processing models trained from a large number of photos of non-specific portraits to enhance the current human image. The features of the human figure in the processed image may include features that do not belong to the current human figure, for example, deep eye sockets and a high nose bridge that do not belong to the current portrait are added.
因此，如何提供一种图像处理方法改善图像处理效果，成为了技术领域内重要的研究课题。Therefore, how to provide an image processing method to improve the image processing effect has become an important research topic in the technical field.
发明内容Summary of the invention
本发明提供一种图像处理方法及电子设备，基于当前被摄人物的历史先验数据中的优质图像对当前待增强人物图像进行增强处理，改善处理后的人物图像中人物的特点包含与当前人物的特点不符的问题。The present invention provides an image processing method and electronic equipment, which enhance the current image of a person to be enhanced based on the high-quality image in the historical prior data of the current person being photographed, and improve the problem that the characteristics of the person in the processed person image do not match the characteristics of the current person.
第一方面，本申请提供一种图像处理方法，所述方法包括：获取第一图像，所述第一图像包含肖像图像；获取与所述第一图像匹配的优质图像，所述优质图像来源于目标人物且所述第一图像中包含所述目标人物的肖像图像，所述优质图像的图像质量优于所述第一图像的图像质量；利用所述优质图像对所述第一图像中的肖像图像的整体图像和/或局部图像进行图像增强处理，得到与所述第一图像对应的第二图像，且所述第二图像的图像质量优于所述第一图像。In a first aspect, the present application provides an image processing method, comprising: acquiring a first image, the first image including a portrait image; acquiring a high-quality image matching the first image, the high-quality image originating from a target person and the first image including a portrait image of the target person, the image quality of the high-quality image being better than the image quality of the first image; using the high-quality image to perform image enhancement processing on an overall image and/or a local image of the portrait image in the first image to obtain a second image corresponding to the first image, and the image quality of the second image being better than that of the first image.
示例性的，所述优质图像的图像质量优于所述第一图像的图像质量包括：所述优质图像的面部特征优于所述第一图像的面部图像，和/或，所述优质图像的五官特征优于所述第一图像的五官特征。例如，该优质图像的整体图像或局部图像的画质较清晰、面部区域或五官区域清晰可见。Exemplarily, the image quality of the high-quality image is better than the image quality of the first image, including: the facial features of the high-quality image are better than the facial image of the first image, and/or the facial features of the high-quality image are better than the facial features of the first image. For example, the overall image or partial image quality of the high-quality image is clearer, and the facial area or facial features area is clearly visible.
示例性的，该优质图像可以为与第一图像的人脸聚类标签一致的全部人物图像中图像综合质量最高的人物图像。Exemplarily, the high-quality image may be a person image with the highest comprehensive image quality among all person images having the same face clustering label as the first image.
在本申请中，优质图像是指电子设备对应的图像数据库中存储的历史图像，该历史图像为在获取到上述第一图像之前获取到的图像。In the present application, a high-quality image refers to a historical image stored in an image database corresponding to the electronic device, and the historical image is an image acquired before the above-mentioned first image is acquired.
上述优质图像与上述第一图像来源于相同人物(例如均来源于上述目标人物)，例如该优质图像的人脸聚类标签与第一图像中对应的肖像的人脸聚类标签相同。The high-quality image and the first image are from the same person (for example, both are from the target person), for example, the face clustering label of the high-quality image is the same as the face clustering label of the corresponding portrait in the first image.
在一种可能的实现方式中，上述优质图像为未进行过图像增强处理的历史图像。In a possible implementation, the high-quality image is a historical image that has not been subjected to image enhancement processing.
例如，优质图像包括优质肖像，对第一图像中的肖像整体图像进行图像增强处理可以包括但不限于：利用优质肖像对第一图像的肖像图像进行增强处理。其中，第一图像的肖像图像可以基于人脸抠图技术获得。For example, the high-quality image includes a high-quality portrait, and the image enhancement processing of the overall portrait image in the first image may include but is not limited to: using the high-quality portrait to enhance the portrait image of the first image. The portrait image of the first image may be obtained based on the face cutout technology.
例如，优质图像包括优质肖像和优质五官图像，对第一图像中的肖像整体图像和肖像局部图像进行图像增强处理可以包括但不限于：利用优质肖像对第一图像的肖像的整体图像进行增强处理，以及利用优质五官图像对增强后的肖像的五官图像进行单独增强处理。For example, a high-quality image includes a high-quality portrait and a high-quality facial feature image. Image enhancement processing of the overall portrait image and the partial portrait image in the first image may include but is not limited to: using the high-quality portrait to enhance the overall image of the portrait of the first image, and using the high-quality facial feature image to separately enhance the facial feature image of the enhanced portrait.
例如，优质图像包括优质面部图像和优质五官图像。对第一图像中的肖像局部图像进行图像增强处理可以包括但不限于：对第一图像中的面部图像进行增强处理，具体可以为利用优质面部对第一图像的面部皮肤进行增强处理，以及对第一图像中的五官图像进行增强处理。其中，第一图像的面部图像、五官图像可以基于人脸解析技术获得。For example, a high-quality image includes a high-quality facial image and a high-quality facial features image. The image enhancement processing of the portrait partial image in the first image may include but is not limited to: enhancing the facial image in the first image, specifically, enhancing the facial skin of the first image using the high-quality face, and enhancing the facial features image in the first image. The facial image and the facial features image of the first image may be obtained based on face analysis technology.
在另外一些可能的实现方式中，对第一图像中的肖像整体图像和/或肖像局部图像进行图像增强处理还可以包括但不限于：对第一图像中的头发或脖子区域对应的图像进行图像增强处理。In some other possible implementations, performing image enhancement processing on the overall portrait image and/or the partial portrait image in the first image may also include but is not limited to: performing image enhancement processing on the image corresponding to the hair or neck area in the first image.
示例性的，可以基于机器学习方法利用上述优质图像对第一图像中的肖像整体图像和/或肖像局部图像进行图像增强处理，机器学习方法可以是构建深度学习网络训练得到模型的方法以及区别于深度学习的传统方法。具体可以是，通过大量特定人物图像数据对(优质图像数据-待增强图像数据)训练深度网络得到深度模型，训练数据可以是整张肖像对、面部皮肤对、五官区域对，该网络可以共同实现利用优质肖像图像对待增强肖像图像增强、利用优质面部皮肤图像对待增强皮肤图像进行增强、利用优质五官图像对带增强五官图像进行增强处理；也可以训练三个模型分别对肖像图像、面部图像、五官图像进行增强处理；也可以利用传统方法，对优质图像和待增强图形进行图像特征融合，图像特征可以是图像的纹理信息、图像的清晰度、图像的色彩信息，传统方法包括传统的图像融合方法泊松融合法、拉普拉斯融合法、局域均值的图像融合方法等。通过深度模型或者传统的方法均可以增强第一图像的面部纹理信息、五官细节以及整体图像的画质清晰度。Exemplarily, the above-mentioned high-quality image can be used to perform image enhancement processing on the portrait overall image and/or the portrait partial image in the first image based on a machine learning method. The machine learning method can be a method of constructing a deep learning network training to obtain a model and a traditional method different from deep learning. Specifically, it can be that a deep network is trained by a large number of specific character image data pairs (high-quality image data-image data to be enhanced) to obtain a deep model. The training data can be a whole portrait pair, a facial skin pair, and a facial feature area pair. The network can jointly realize the enhancement of the portrait image to be enhanced by using the high-quality portrait image, the enhancement of the skin image to be enhanced by using the high-quality facial skin image, and the enhancement of the facial feature image with enhanced facial features by using the high-quality facial features image; three models can also be trained to enhance the portrait image, facial image, and facial feature image respectively; image features can also be fused by traditional methods for the high-quality image and the graphics to be enhanced. The image features can be the texture information of the image, the clarity of the image, and the color information of the image. The traditional methods include traditional image fusion methods such as Poisson fusion method, Laplace fusion method, and local mean image fusion method. The facial texture information, facial features details, and the image quality clarity of the overall image of the first image can be enhanced by either the deep model or the traditional method.
可理解的，若采用由大量非特定人像的照片训练得到的图像处理模型对当前人物图像进行增强处理，处理后的人物图像中人物的特点可能会包含不属于当前人物的特点，例如增加了并不属于当前人像的深邃的眼窝、高挺的鼻子山根等，又例如包含不属于当前人物的皮肤纹理，使得处理后的人物图像的肤质与被摄人物的肤质存在较大差异的问题等。It is understandable that if an image processing model trained by a large number of photos of non-specific portraits is used to enhance the current person image, the characteristics of the person in the processed person image may include characteristics that do not belong to the current person, such as the addition of deep eye sockets and a high nose bridge that do not belong to the current portrait, or the inclusion of skin texture that does not belong to the current person, resulting in a significant difference in the skin quality of the processed person image and the skin quality of the photographed person.
然而采用本申请实施例提供的图像处理方法，一方面，上述优质图像的肖像和第一图像中的肖像是相同人物，且优质图像的肖像全局和/或局部清晰度和皮肤纹理细节的细腻度优于第一图像，采用该优质图像对第一图像的肖像整体图像和/或肖像局部图像进行图像增强，可以增强第一图像的面部纹理信息，增强第一图像的五官细节信息、整体图像的画质清晰度，而不会引入不是该人物的特征。例如基于该优质图像对第一图像的肖像整体图像和/或肖像局部图像进行增强处理包括但不限于：面部纹理细节的增强、嘴唇纹理细节的增强、眉毛对比度的增强(以使得增强眉毛区域的毛流感细节)、鼻子区域色彩对比度的增强(以使得提高鼻子区域对应图像的几何立体度)、耳朵清晰的增强、眼神光的光斑轮廓清晰度和光斑亮度的增强、眼睛虹膜色彩信息和虹膜纹理的增强。并且该面部纹理细节、五官细节基于相同的被摄对象的优质图像增强实现的，从而增强得到的上述第一增强图像和第二图像不会存在处理后的人物图像中人物的特点包含不属于当前人物的特点的问题。另外一方面，在丰富第一图像的纹理细节的同时，以好的面部纹理细节和五官细节解决肖像的整体图像清晰度、人像噪声、纹理细节缺失或涂抹等的问题。However, using the image processing method provided in the embodiment of the present application, on the one hand, the portrait in the above-mentioned high-quality image and the portrait in the first image are of the same person, and the global and/or local clarity and fineness of the skin texture details of the portrait in the high-quality image are better than those of the first image. Using the high-quality image to enhance the overall image and/or the partial image of the portrait of the first image can enhance the facial texture information of the first image, enhance the facial features details information of the first image, and enhance the image quality clarity of the overall image without introducing features that are not of the person. For example, based on the high-quality image, the enhancement processing of the overall image and/or the partial image of the portrait of the first image includes but is not limited to: enhancement of facial texture details, enhancement of lip texture details, enhancement of eyebrow contrast (so as to enhance the hair flow details of the eyebrow area), enhancement of nose area color contrast (so as to improve the geometric three-dimensionality of the image corresponding to the nose area), enhancement of ear clarity, enhancement of the spot contour clarity and spot brightness of the eye light, and enhancement of eye iris color information and iris texture. Furthermore, the facial texture details and facial features details are enhanced based on the same high-quality image of the subject, so that the first enhanced image and the second enhanced image obtained by the enhancement will not have the problem that the features of the person in the processed person image include features that do not belong to the current person. On the other hand, while enriching the texture details of the first image, the good facial texture details and facial features details can solve the problems of overall image clarity, portrait noise, missing or smeared texture details, etc.
采用本申请提供的方法可以解决在一些场景下，例如包括但不限于暗光场景、欠曝或过曝场景、拍照抖动场景下拍摄得到的人物图像的图像模糊、噪点多、纹理细节缺失或涂抹、光照不足、光照过亮的问题，以及在增强图像质量的同时改善处理后的人物图像中人物的特点包含不属于当前人物的特点的问题。The method provided in the present application can solve the problems of blurred image, high noise, missing or smeared texture details, insufficient lighting, and excessive lighting in some scenarios, such as but not limited to dark scenes, underexposure or overexposure scenes, and camera shaking scenes, as well as the problem that the characteristics of the person in the processed person image include characteristics that do not belong to the current person while enhancing the image quality.
在一种可能的实现方式中，所述优质图像的图像质量优于所述第一图像的图像质量包括：所述优质图像的全局或局部清晰度、人脸皮肤纹理、以及五官细节优于所述第一图像，所述五官细节包括以下一项或一项以上：五官立体感、嘴唇纹理、眼神光的光斑清晰度、眼睛睁开程度、嘴巴张开程度；所述第二图像的图像质量优于所述第一图像包括：所述第二图像的画质清晰度、面部清晰度、五官清晰度、人脸皮肤纹理、五官立体感、嘴唇纹理、眉毛区域的对比度反映的毛流感、眼睛区域中包含的眼神光的光斑轮廓清晰度、眼神光的光斑亮度、虹膜区域的纹理、虹膜区域的色彩中的一项或一项以上优于所述第一图像。In a possible implementation, the image quality of the high-quality image is better than that of the first image, including: the global or local clarity, facial skin texture, and facial features details of the high-quality image are better than those of the first image, and the facial features details include one or more of the following: three-dimensional sense of facial features, lip texture, clarity of the spot of eye light, degree of eye openness, degree of mouth openness; the image quality of the second image is better than that of the first image, including: one or more of the picture quality clarity, facial clarity, facial features clarity, facial skin texture, three-dimensional sense of facial features, lip texture, hair flow reflected by the contrast of the eyebrow area, clarity of the spot outline of the eye light contained in the eye area, brightness of the spot of the eye light, texture of the iris area, and color of the iris area of the second image are better than those of the first image.
在一种可能的实现方式中，所述利用所述优质图像对所述第一图像中的肖像图像的整体图像和/或局部图像进行图像增强处理，得到与所述第一图像对应的第二图像，包括：将所述优质图像和所述第一图像输入图像增强模型，通过所述图像增强模型利用所述优质图像对所述第一图像的肖像整体图像和/或肖像局部图像进行增强处理，得到所述第二图像，所述图像增强模型基于机器学习中的有监督学习方法或无监督学习方法训练得到。In a possible implementation, the method of using the high-quality image to perform image enhancement processing on the overall image and/or the local image of the portrait image in the first image to obtain a second image corresponding to the first image includes: inputting the high-quality image and the first image into an image enhancement model, and using the high-quality image through the image enhancement model to perform image enhancement processing on the overall portrait image and/or the local portrait image of the first image to obtain the second image, wherein the image enhancement model is trained based on a supervised learning method or an unsupervised learning method in machine learning.
在一种可能的实现方式中，所述优质图像至少包含面部图像和五官图像，所述优质图像为一张图像中的肖像整体图像，所述肖像整体图像包含面部图像和五官图像；或者，所述优质图像为人脸不同部位对应的局部图像的图像集合、且所述优质图像中的每个五官区域来源于同一张图像或来源于不同图像；所述利用所述优质图像对所述第一图像中的肖像图像的整体图像和/或局部图像进行图像增强处理，包括：基于所述优质图像中的面部图像对所述第一图像中的面部图像的面部纹理进行增强处理，和/或，基于所述优质图像中的五官图像对所述第一图像中对应的五官图像进行增强处理。In a possible implementation, the high-quality image includes at least a facial image and a facial feature image, and the high-quality image is an overall portrait image in one image, and the overall portrait image includes a facial image and a facial feature image; or, the high-quality image is an image collection of local images corresponding to different parts of the face, and each facial feature area in the high-quality image comes from the same image or from different images; using the high-quality image to perform image enhancement processing on the overall image and/or local image of the portrait image in the first image includes: enhancing the facial texture of the facial image in the first image based on the facial image in the high-quality image, and/or enhancing the corresponding facial feature images in the first image based on the facial feature images in the high-quality image.
示例性的，上述优质图像中的面部图像的图像质量优于第一图像的面部图像的图像质量，上述优质图像中的五官图像的图像质量优于第一图像的面部图像的图像质量。例如优质图像的面部图像的清晰度、人脸皮肤纹理优于第一图像的面部图像，具体的优质图像的面部图像的皮肤纹理细腻度和质感优于第一图像的面部图像。优质图像中的五官图像的五官立体感、嘴唇纹理细节、眼神光的光斑清晰度、眼睛睁开程度、嘴巴张开程度优于第一图像的五官图像。Exemplarily, the image quality of the facial image in the above-mentioned high-quality image is better than the image quality of the facial image in the first image, and the image quality of the facial features image in the above-mentioned high-quality image is better than the image quality of the facial image in the first image. For example, the clarity of the facial image and the skin texture of the face in the high-quality image are better than those of the facial image in the first image, and specifically, the skin texture fineness and texture of the facial image in the high-quality image are better than those of the facial image in the first image. The three-dimensional sense of the facial features, the details of the lip texture, the clarity of the spot of the eye light, the degree of eye openness, and the degree of mouth openness of the facial features image in the high-quality image are better than those of the facial features image in the first image.
在一种可能的实现方式中，所述基于所述优质图像中的五官图像对所述第一图像中对应的五官图像进行增强处理包括以下至少一项：基于所述优质图像中的嘴巴图像对所述第一图像的嘴巴图像中的嘴唇纹理和色彩进行增强处理；在所述第一图像的眼睛图像中包含眼神光的情况下，对所述眼神光的光斑轮廓清晰度和/或眼神光的光斑亮度进行增强处理；基于第一参考虹膜图像对所述第一图像的眼睛部位的虹膜进行虹膜纹理和虹膜色彩的增强处理，所述第一参考虹膜图像与目标虹膜图像的相似度满足预设虹膜相似度条件，所述目标虹膜图像为所述第一图像中的虹膜图像，所述第一参考虹膜图像的色彩和纹理优于所述目标虹膜图像。In a possible implementation, the enhancing processing of the corresponding facial feature images in the first image based on the facial feature images in the high-quality image includes at least one of the following: enhancing the lip texture and color in the mouth image of the first image based on the mouth image in the high-quality image; when the eye image of the first image includes a catch light, enhancing the clarity of the spot contour of the catch light and/or the brightness of the spot of the catch light; enhancing the iris texture and iris color of the iris in the eye part of the first image based on a first reference iris image, the similarity between the first reference iris image and the target iris image meets a preset iris similarity condition, the target iris image is the iris image in the first image, and the color and texture of the first reference iris image are better than those of the target iris image.
在一种可能的实现方式中，上述目标虹膜图像也可以为利用优质肖像对第一图像进行图像增强后的到的第一增强肖像中的虹膜图像。In a possible implementation manner, the target iris image may also be an iris image in a first enhanced portrait obtained by performing image enhancement on the first image using a high-quality portrait.
在一种可能的实现方式中，所述基于所述优质图像中的五官图像对所述第一图像中对应的五官图像进行增强处理还包括以下至少一项：基于所述优质图像中的眼睛图像对所述第一图像的眼睛图像的进行清晰度增强处理；基于所述优质图像中的鼻子图像对所述第一图像的鼻子图像的清晰度和立体度进行增强处理；基于所述优质图像中的眉毛图像对所述第一图像的眉毛图像的清晰度和毛流感进行增强处理；基于所述优质图像中的耳朵图像对所述第一图像的耳朵图像的清晰度进行增强处理。In a possible implementation, the enhancing processing of the corresponding facial feature images in the first image based on the facial feature images in the high-quality image also includes at least one of the following: enhancing the clarity of the eye image of the first image based on the eye image in the high-quality image; enhancing the clarity and stereoscopicness of the nose image of the first image based on the nose image in the high-quality image; enhancing the clarity and hair flow of the eyebrow image of the first image based on the eyebrow image in the high-quality image; enhancing the clarity of the ear image of the first image based on the ear image in the high-quality image.
在本申请实施例中，分别利用优质五官图像中的五官特征对第一图像的肖像图像中的不同五官图像进行五官增强，可以考量到脸部不同区域的细节特征的差异，增强后得到的图像真实感更好。In an embodiment of the present application, facial features in high-quality facial feature images are used to perform facial enhancement on different facial feature images in the portrait image of the first image, respectively. Differences in detail features of different facial regions can be taken into account, and the enhanced image has a better sense of reality.
示例性的，所述优质图像为属于肖像整体图像的优质肖像，所述优质肖像的图像质量优于所述第一图像的肖像图像的图像质量，所述利用所述优质图像对所述第一图像中的肖像图像的整体图像和/或局部图像进行图像增强处理，得到与所述第一图像对应的第二图像，包括：利用所述优质肖像对所述第一图像的肖像图像的整体图像进行图像增强处理，得到所述第二图像。或者也可以理解为，所述基于所述优质图像中的面部图像对所述第一图像中的面部图像的面部纹理进行增强处理，和/或，基于所述优质图像中的五官图像对所述第一图像中对应的五官图像进行增强处理，具体包括：利用所述优质肖像对所述第一图像的肖像图像的整体图像进行图像增强处理，得到所述第二图像。Exemplarily, the high-quality image is a high-quality portrait belonging to the overall image of the portrait, and the image quality of the high-quality portrait is better than the image quality of the portrait image of the first image. The use of the high-quality image to perform image enhancement processing on the overall image and/or partial image of the portrait image in the first image to obtain a second image corresponding to the first image includes: using the high-quality portrait to perform image enhancement processing on the overall image of the portrait image of the first image to obtain the second image. Alternatively, it can also be understood that the facial texture of the facial image in the first image is enhanced based on the facial image in the high-quality image, and/or the facial features image corresponding to the first image is enhanced based on the facial features image in the high-quality image, specifically including: using the high-quality portrait to perform image enhancement processing on the overall image of the portrait image of the first image to obtain the second image.
在一种可能的实现方式中，所述优质图像包含肖像整体图像的优质肖像，则相应地，该优质肖像对应的肖像清晰度优于待增强肖像的肖像清晰度，以及该优质肖像的面部图像纹理、清晰度和/或五官图像色彩及清晰度优于待增强肖像中对应区域的清晰度，该优质肖像的对应的整体图像包含该优质肖像和该优质肖像对应的背景图像，该待增强肖像的整体图像包含该待增强肖像和上述原始背景图像。In a possible implementation, the high-quality image includes a high-quality portrait of the overall image of the portrait. Accordingly, the portrait clarity corresponding to the high-quality portrait is better than the portrait clarity of the portrait to be enhanced, and the facial image texture, clarity and/or facial features image color and clarity of the high-quality portrait are better than the clarity of the corresponding area in the portrait to be enhanced. The overall image corresponding to the high-quality portrait includes the high-quality portrait and the background image corresponding to the high-quality portrait, and the overall image of the portrait to be enhanced includes the portrait to be enhanced and the above-mentioned original background image.
在一种可能的实现方式中，所述优质图像包含肖像整体图像的优质肖像，所述优质图像的图像质量优于所述第一图像的图像质量包括：所述优质肖像的画质清晰度、面部清晰度、五官清晰度、人脸皮肤纹理、五官立体感、以及嘴唇纹理优于所述第一图像的肖像图像。In a possible implementation, the high-quality image includes a high-quality portrait of the overall portrait image, and the image quality of the high-quality image is better than that of the first image, including: the image clarity, facial clarity, facial feature clarity, facial skin texture, three-dimensional sense of facial features, and lip texture of the high-quality portrait are better than those of the portrait image of the first image.
在一种可能的实现方式中，所述利用所述优质肖像对所述第一图像的肖像图像的整体图像进行图像增强处理，得到所述第二图像，包括：将所述优质肖像和所述第一图像的肖像图像输入肖像增强模型，通过所述肖像增强模型基于所述优质肖像对所述第一图像的肖像图像进行肖像增强，得到所述第二图像；所述肖像增强模型采用有监督学习方法或无监督学习方法训练得到；其中，在采用有监督学习方法和至少两组实验数据训练得到所述肖像增强模型的情况下，每一组所述实验数据包括第一样本图像、第二样本图像、以及第三样本图像，所述第一样本图像和所述第二样本图像作为所述肖像增强模型的输入，所述第三样本图像作为第一输出图像的约束图像，所述第一输出图像为所述第一肖像增强模型中输出的与所述第一样本图像对应的增强处理后的图像，所述第一样本图像和所述第三样本图像包含相同肖像的两张不同图像质量的肖像，所述第三样本图像的图像质量优于所述第一样本图像的图像质量，所述第二样本图像包含第一参考肖像、且所述第二样本图像的图像质量优于所述第一样本图像，所述第一样本图像包含的肖像图像与所述第一参考肖像对应同一人物；或者，在采用无监督学习方法和至少两组实验数据训练得到所述肖像增强模型的情况下，每一组所述实验数据包括第四样本图像和第五样本图像，所述肖像增强模型包括生成模型和对抗模型，所述生成模型用于以所述第五样本图像作为指导图像生成与所述第四样本图像对应的增强处理后的第二输出图像，所述对抗模型为预训练好的用于评判所述第二输出图像是否符合增强效果的评判网络，所述第五样本图像包含第二参考肖像、且所述第五样本图像的图像质量优于所述第四样本图像，所述第四样本图像包含的肖像图像与所述第二参考肖像对应同一人物。In a possible implementation, the using of the high-quality portrait to perform image enhancement processing on the overall image of the portrait image of the first image to obtain the second image includes: inputting the high-quality portrait and the portrait image of the first image into a portrait enhancement model, and performing portrait enhancement processing on the portrait image of the first image based on the high-quality portrait by the portrait enhancement model to obtain the second image; the portrait enhancement model is trained by a supervised learning method or an unsupervised learning method; wherein, in the case of training the portrait enhancement model by a supervised learning method and at least two groups of experimental data, each group of the experimental data includes a first sample image, a second sample image, and a third sample image, the first sample image and the second sample image are used as inputs of the portrait enhancement model, the third sample image is used as a constraint image of a first output image, the first output image is an enhanced image corresponding to the first sample image outputted from the first portrait enhancement model, and the first sample image and the third sample image contain the same portrait two portraits of different image qualities, the image quality of the third sample image is better than that of the first sample image, the second sample image includes a first reference portrait, and the image quality of the second sample image is better than that of the first sample image, and the portrait image included in the first sample image corresponds to the same person as the first reference portrait; or, in the case where the portrait enhancement model is obtained by training with an unsupervised learning method and at least two groups of experimental data, each group of the experimental data includes a fourth sample image and a fifth sample image, the portrait enhancement model includes a generative model and an adversarial model, the generative model is used to generate a second output image after enhancement corresponding to the fourth sample image using the fifth sample image as a guide image, the adversarial model is a pre-trained judgment network for judging whether the second output image meets the enhancement effect, the fifth sample image includes a second reference portrait, and the image quality of the fifth sample image is better than that of the fourth sample image, and the portrait image included in the fourth sample image corresponds to the same person as the second reference portrait.
示例性的，所述无监督学习方法包括生成对抗网络GAN、流归一化网络FlowNormalization、扩散模型DiffusionModel。Exemplarily, the unsupervised learning method includes a generative adversarial network GAN, a flow normalization network FlowNormalization, and a diffusion model DiffusionModel.
在一种可能的实现方式中，所述利用所述优质肖像对所述第一图像的肖像图像的整体图像进行图像增强处理，得到所述第二图像，包括：利用所述优质肖像对所述第一图像的中的肖像图像的整体图像进行增强处理，得到第一增强肖像；以及，对所述第一增强肖像进行以下一项或一项以上增强处理得到所述第二图像：基于优质嘴唇图像对所述第一增强肖像中的嘴唇图像的色彩和纹理进行增强处理，所述优质嘴唇图像与所述第一图像来源于相同人物，所述优质嘴唇图像的图像质量优于所述第一图像的嘴唇图像的图像质量；在所述第一增强肖像中的眼睛中包含眼神光的情况下，对所述第一增强肖像的眼神光的光斑轮廓清晰度和/或眼神光的光斑亮度进行增强处理；基于优质鼻子图像对所述第一增强肖像中的鼻子图像的立体度进行增强处理，所述优质鼻子图像与所述第一图像来源于相同人物，所述优质鼻子图像的图像质量优于所述第一图像的鼻子图像的图像质量；基于优质眉毛图像对所述第一增强肖像中的眉毛图像的毛流感进行增强处理，所述优质眉毛图像与所述第一图像来源于相同人物，所述优质眉毛图像的图像质量优于所述第一图像的眉毛图像的图像质量；基于第一参考虹膜图像对所述第一增强肖像的眼睛部位的虹膜进行虹膜纹理和虹膜色彩的增强处理，所述第一参考虹膜图像与目标虹膜图像的相似度满足预设虹膜相似度条件，所述目标虹膜图像为所述第一图像中的虹膜图像或为所述第一增强肖像中的虹膜图像，所述第一参考虹膜图像的色彩和纹理优于所述目标虹膜图像；其中，所述优质嘴唇图像、所述优质鼻子图像、以及所述优质眉毛图像来源于同一张图像或不同图像。In a possible implementation, the using the high-quality portrait to perform image enhancement processing on the overall image of the portrait image of the first image to obtain the second image includes: using the high-quality portrait to enhance the overall image of the portrait image in the first image to obtain a first enhanced portrait; and performing one or more of the following enhancement processing on the first enhanced portrait to obtain the second image: enhancing the color and texture of the lip image in the first enhanced portrait based on a high-quality lip image, the high-quality lip image and the first image are from the same person, and the image quality of the high-quality lip image is better than the image quality of the lip image in the first image; in the case where the eyes in the first enhanced portrait contain eye light, enhancing the spot contour clarity and/or the spot brightness of the eye light of the first enhanced portrait; enhancing the stereoscopic degree of the nose image in the first enhanced portrait based on the high-quality nose image, The high-quality nose image and the first image are from the same person, and the image quality of the high-quality nose image is better than the image quality of the nose image of the first image; the hair flow of the eyebrow image in the first enhanced portrait is enhanced based on the high-quality eyebrow image, the high-quality eyebrow image and the first image are from the same person, and the image quality of the high-quality eyebrow image is better than the image quality of the eyebrow image of the first image; the iris texture and iris color of the iris in the eye part of the first enhanced portrait are enhanced based on the first reference iris image, the similarity between the first reference iris image and the target iris image meets the preset iris similarity condition, the target iris image is the iris image in the first image or the iris image in the first enhanced portrait, and the color and texture of the first reference iris image are better than those of the target iris image; wherein the high-quality lip image, the high-quality nose image, and the high-quality eyebrow image are from the same image or different images.
在对待增强肖像进行肖像整体增强后，再利用对应的优质五官图像对第一增强肖像进行五官单独增强处理，可以进一步提高五官图像的图像效果。After the portrait to be enhanced is enhanced as a whole, the corresponding high-quality facial feature images are used to perform individual facial feature enhancement processing on the first enhanced portrait, which can further improve the image effect of the facial feature images.
上述基于第一参考虹膜图像对所述第一增强肖像的眼睛部位的虹膜进行虹膜纹理和虹膜色彩的增强处理具体可以是：从虹膜信息库中查找与该待增强肖像或该第一增强图像的虹膜的色彩相似度较高的上述第一参考虹膜图像，基于深度学习方法和该第一参考虹膜图像的虹膜纹理信息和色彩信息，对第一增强肖像的眼睛部位的虹膜进行虹膜纹理和虹膜色彩的增强处理。由此，在改善处理得到的人物图像的肤质与被摄人物的肤质存在较大差异的问题的同时，对眼部虹膜进行纹理细节填充，进一步改善增强后的图像(也即第二图像)在各个图像区域的清晰度的一致性，以及进一步改善增强后的图像的真实感。The above-mentioned enhancement of the iris texture and iris color of the iris at the eye part of the first enhanced portrait based on the first reference iris image may specifically be: searching the above-mentioned first reference iris image with a high color similarity to the iris of the portrait to be enhanced or the first enhanced image from the iris information database, and enhancing the iris texture and iris color of the iris at the eye part of the first enhanced portrait based on the deep learning method and the iris texture information and color information of the first reference iris image. Thus, while improving the problem that the skin quality of the processed person image is greatly different from the skin quality of the photographed person, the texture details of the eye iris are filled, and the consistency of the clarity of the enhanced image (i.e., the second image) in each image area is further improved, and the realism of the enhanced image is further improved.
在一种可能的实现方式中，所述优质图像包括优质面部图像和优质五官图像，所述优质面部图像包含人脸中的面部区域对应的局部图像，所述优质五官图像包含人脸中五官不同部位对应的局部图像，所述优质面部图像的图像质量优于所述第一图像的面部图像的图像质量，所述优质五官图像的图像质量优于所述第一图像的五官图像的图像质量，所述优质五官图像中的每个五官区域来源于同一张图像或来源于不同图像，所述利用所述优质图像对所述第一图像中的肖像图像的整体图像和/或局部图像进行图像增强处理，包括：利用所述优质五官图像对所述第一图像的五官图像进行五官增强处理，以及，利用所述优质面部图像对所述第一图像的面部图像进行面部增强处理。In a possible implementation, the high-quality image includes a high-quality facial image and a high-quality facial features image, the high-quality facial image includes a local image corresponding to a facial area in a human face, the high-quality facial features image includes local images corresponding to different parts of the facial features in a human face, the image quality of the high-quality facial image is better than the image quality of the facial image of the first image, the image quality of the high-quality facial features image is better than the image quality of the facial features image of the first image, each facial feature area in the high-quality facial features image comes from the same image or from different images, and using the high-quality image to perform image enhancement processing on the overall image and/or local image of the portrait image in the first image includes: using the high-quality facial features image to perform facial features enhancement processing on the facial features image of the first image, and using the high-quality facial image to perform facial enhancement processing on the facial image of the first image.
或者，也可以理解为，上述基于所述优质图像中的面部图像对所述第一图像中的面部图像的面部纹理进行增强处理，和/或，基于所述优质图像中的五官图像对所述第一图像中对应的五官图像进行增强处理，具体包括：利用所述优质五官图像对所述第一图像的五官图像进行五官增强处理，以及，利用所述优质面部图像对所述第一图像的面部图像进行面部增强处理。Alternatively, it can also be understood that the above-mentioned enhancing processing of the facial texture of the facial image in the first image based on the facial image in the high-quality image, and/or enhancing processing of the corresponding facial feature images in the first image based on the facial feature images in the high-quality image, specifically includes: performing facial feature enhancement processing on the facial feature images of the first image using the high-quality facial feature images, and performing facial enhancement processing on the facial image of the first image using the high-quality facial image.
可理解的，相比于利用优质肖像对第一图像的肖像图像进行整体增强的方案，分别利用优质面部图像对第一图像的面部图像进行增强处理以及利用优质五官图像对第一图像的五官图像进行增强处理，可以更好地考量到脸部不同区域的几何差异、细节特征差异，图像增强效果可能会更好，增强后得到的图像真实感可能会更好。It is understandable that compared with the solution of using a high-quality portrait to perform overall enhancement on the portrait image of the first image, separately enhancing the facial image of the first image using a high-quality facial image and enhancing the facial image of the first image using a high-quality facial feature image can better take into account the geometric differences and detail feature differences in different facial regions, the image enhancement effect may be better, and the image realism obtained after enhancement may be better.
在一种可能的实现方式中，所述利用所述优质五官图像对所述第一图像的五官图像进行五官增强处理，包括：基于深度学习方法和/或传统方法利用所述优质五官图像对所述第一图像的五官图像进行五官增强处理；其中，基于深度学习方法利用所述优质五官图像对所述第一图像的五官图像进行五官增强处理包括：将所述优质五官图像和所述第一图像的五官图像输入五官增强模型，基于有监督学习方法或无监督学习方法在所述五官增强模型中利用所述优质五官图像对所述第一图像的五官图像进行五官增强处理；所述五官增强模型为集成网络，所述集成网络用于利用所述优质五官图像对所述第一图像的五官图像进行五官增强处理；或者，所述五官增强模型包括至少两个子网络，所述子网络用于利用所述优质五官图像中的子五官图像分别对所述第一图像的五官图像中对应的子五官图像进行五官增强处理，所述子五官图像包括嘴唇图像、鼻子图像、眼睛图像、眉毛图像、以及耳朵图像中的至少一项；其中，基于传统方法利用所述优质五官图像对所述第一图像的五官图像进行五官增强处理包括：提取优质五官图像中一个或一个以上五官图像体现的特征，基于提取到的特征信息进行特征拟合计算，以便于将优质五官图像中的五官局部图像的优质特征融合到第一图像的五官图像中对应的五官局部图像中。In a possible implementation, the using of the high-quality facial feature image to perform facial feature enhancement processing on the facial feature image of the first image includes: using the high-quality facial feature image to perform facial feature enhancement processing on the facial feature image of the first image based on a deep learning method and/or a traditional method; wherein, using the high-quality facial feature image to perform facial feature enhancement processing on the facial feature image of the first image based on a deep learning method includes: inputting the high-quality facial feature image and the facial feature images of the first image into a facial feature enhancement model, and using the high-quality facial feature image in the facial feature enhancement model to perform facial feature enhancement processing on the facial feature image of the first image based on a supervised learning method or an unsupervised learning method; the facial feature enhancement model is an integrated network, and the integrated network is used to use the high-quality facial feature image to perform facial feature enhancement processing on the facial feature image of the first image The facial feature enhancement model comprises at least two sub-networks, wherein the sub-networks are used to respectively perform facial feature enhancement processing on the corresponding sub-facial feature images in the facial feature image of the first image using the sub-facial feature images in the high-quality facial feature image, wherein the sub-facial feature images include at least one of a lip image, a nose image, an eye image, an eyebrow image, and an ear image; wherein the facial feature enhancement processing on the facial feature image of the first image using the high-quality facial feature image based on the traditional method comprises: extracting features embodied in one or more facial feature images in the high-quality facial feature image, and performing feature fitting calculation based on the extracted feature information, so as to fuse the high-quality features of the facial feature local images in the high-quality facial feature image into the corresponding facial feature local images in the facial feature image of the first image.
在一种可能的实现方式中，所述优质图像包含肖像整体图像的优质肖像，在本申请实施例中，优质肖像的获取方法有以下两种，方式1：所述获取与所述第一图像匹配的优质图像包括：确定第一肖像集合，所述第一肖像集合中的每一个肖像图像与所述第一图像来源于相同人物，所述第一肖像集合中的任意一个肖像图像的画质清晰度值满足预设画质清晰度阈值，所述第一肖像集合中任意一个肖像图像的肖像综合评分结果优于所述第一图像的肖像综合评分结果；确定所述第一肖像集合中肖像综合评分结果最优的肖像图像作为所述优质肖像，所述优质肖像的肖像综合评分结果与以下一项或一项以上数据相关：所述优质肖像的面部清晰度、五官清晰度、人脸皮肤纹理、五官立体感、嘴唇纹理细节。In one possible implementation, the high-quality image includes a high-quality portrait of the overall portrait image. In an embodiment of the present application, there are two methods for obtaining a high-quality portrait. Method 1: Obtaining a high-quality image matching the first image includes: determining a first portrait set, each portrait image in the first portrait set and the first image are from the same person, the image quality clarity value of any portrait image in the first portrait set meets a preset image quality clarity threshold, and the portrait comprehensive score result of any portrait image in the first portrait set is better than the portrait comprehensive score result of the first image; determining the portrait image with the best portrait comprehensive score result in the first portrait set as the high-quality portrait, and the portrait comprehensive score result of the high-quality portrait is related to one or more of the following data: facial clarity, facial feature clarity, facial skin texture, facial feature three-dimensionality, and lip texture details of the high-quality portrait.
方式2：所述获取与所述第一图像匹配的优质图像包括：确定第一肖像集合，所述第一肖像集合中的每一个肖像图像与所述第一图像来源于相同人物，所述第一肖像集合中的任意一个肖像图像的画质清晰度值满足预设画质清晰度阈值，所述第一肖像集合中任意一个肖像图像的肖像综合评分结果大于所述第一图像的肖像综合评分结果；确定第一肖像图像，所述第一肖像图像为所述第一肖像集合中肖像综合评分结果最优的肖像图像，所述第一肖像图像的肖像综合评分结果与以下一项或一项以上数据相关：所述优质肖像的面部清晰度、五官清晰度、人脸皮肤纹理、五官立体感、嘴唇纹理细节；在确定所述第一肖像图像的头部姿态与所述第一图像的肖像图像的头部姿态的差值大于或等于第一预设差值的情况下，确定是否存在第二肖像图像，所述第二肖像图像的头部姿态与所述第一图像的肖像图像的头部姿态的差值小于第二预设差值、且所述第二肖像图像的肖像综合评分结果包含于第二肖像集合，所述第二肖像集合为所述第一评分集合中肖像综合评分结果由小到大排序的序列中排名靠前预设百分比(例如排名前10％)的评分集合，所述第二预设差值小于或等于所述第一预设差值；在确定存在所述第二肖像图像的情况下，确定第一参考图像集合中与所述第一图像的头部姿态的差值最小的肖像图像作为所述优质图像，第一参考图像集合包括至少一个所述第二肖像图像。Method 2: The acquisition of a high-quality image matching the first image includes: determining a first portrait set, wherein each portrait image in the first portrait set is from the same person as the first image, the image quality clarity value of any portrait image in the first portrait set satisfies a preset image quality clarity threshold, and the portrait comprehensive score result of any portrait image in the first portrait set is greater than the portrait comprehensive score result of the first image; determining a first portrait image, wherein the first portrait image is the portrait image with the best portrait comprehensive score result in the first portrait set, and the portrait comprehensive score result of the first portrait image is related to one or more of the following data: facial clarity, facial feature clarity, facial skin texture, facial feature stereoscopic effect, and lip texture details of the high-quality portrait; and determining the head posture of the first portrait image and the first portrait image; wherein the head posture of the first portrait image and the first portrait image are matched. When the difference in head posture of a portrait image of an image is greater than or equal to a first preset difference, it is determined whether there is a second portrait image, the difference in head posture between the second portrait image and the head posture of the portrait image of the first image is less than the second preset difference, and the comprehensive portrait score result of the second portrait image is included in a second portrait set, the second portrait set is a score set that ranks in a preset percentage (for example, the top 10%) in a sequence of comprehensive portrait score results in the first score set sorted from small to large, and the second preset difference is less than or equal to the first preset difference; when it is determined that the second portrait image exists, the portrait image in the first reference image set with the smallest difference in head posture with the first image is determined as the high-quality image, and the first reference image set includes at least one second portrait image.
例如上述第一预设差值可以为10或20，上述第二预设差值可以为10或20，具体根据需求而定，本文对此不做限定。For example, the first preset difference may be 10 or 20, and the second preset difference may be 10 or 20, depending on specific needs, which is not limited in this document.
可理解的，在选取优质肖像时，若依据肖像图像的质量评分(也即肖像综合评分结果)选取的第一肖像图像的头部姿态与第一图像的头部姿态相差较大的情况下，选取第一肖像集合中排名前预设百分比肖像图像中与第一图像的头部姿态最接近的图像，也即选取一个图像质量较优且与第一图像的头部姿态匹配度较高的图像。不仅考虑肖像图像的质量评分，还考虑历史图像的肖像图像与第一图像的头部姿态的匹配度，得到的优质肖像与第一图像的头部姿态相近，优质肖像包含更为完整的优质脸部信息，为图像增强提供保障。It is understandable that when selecting a high-quality portrait, if the head posture of the first portrait image selected based on the quality score of the portrait image (i.e., the comprehensive portrait score result) is significantly different from the head posture of the first image, an image with the head posture closest to the first image among the portrait images ranked in the top preset percentage in the first portrait set is selected, that is, an image with better image quality and a higher degree of matching with the head posture of the first image is selected. Not only the quality score of the portrait image is considered, but also the degree of matching between the portrait image of the historical image and the head posture of the first image is considered. The obtained high-quality portrait has a similar head posture to the first image, and the high-quality portrait contains more complete high-quality facial information, providing a guarantee for image enhancement.
在一种可能的实现方式中，所述优质肖像的画质清晰度值为目标评分区域对应的清晰度值；其中，在所述优质肖像对应的拍照模式为人像模式的情况下，所述目标评分区域为所述优质肖像中的肖像图像对应的区域，在所述优质肖像对应的拍照模式不为人像模式的情况下，所述目标评分区域为所述优质肖像对应的原始图像的全局图像区域，所述原始图像为与所述优质肖像对应的包含肖像图像和背景图像的图像。In a possible implementation, the image quality clarity value of the high-quality portrait is a clarity value corresponding to a target scoring area; wherein, when the shooting mode corresponding to the high-quality portrait is a portrait mode, the target scoring area is an area corresponding to the portrait image in the high-quality portrait; and when the shooting mode corresponding to the high-quality portrait is not a portrait mode, the target scoring area is a global image area of an original image corresponding to the high-quality portrait, and the original image is an image corresponding to the high-quality portrait and including a portrait image and a background image.
示例性的，根据图像的拍照模式的不同(例如自动模式、人像模式、夜景模式、专业模式等)，画质清晰度的评分区域不同。例如，在进行画质清晰度评分时会读取当前照片的拍照模式，若为人像拍照模式，那么画质清晰度评分区域只对人像区域进行评价，也就是说，若优质肖像对应的整体图像的拍照模式为人像拍照模式，则该优质肖像的画质清晰度评分结果为肖像区域对应的图像的画质清晰度的评分结果(也即局部清晰度评分结果)。若不为人像拍照模式，那么评分区域为整张图像，也就是说，若优质肖像对应的整体图像的拍照模式不为人像拍照模式，优质肖像的画质清晰度评分结果为该优质肖像对应的整体图像的全局画质清晰度的评分结果。Exemplarily, the scoring area for image quality clarity is different depending on the shooting mode of the image (e.g., automatic mode, portrait mode, night scene mode, professional mode, etc.). For example, when scoring image quality clarity, the shooting mode of the current photo will be read. If it is a portrait shooting mode, the image quality clarity scoring area will only evaluate the portrait area. That is, if the shooting mode of the overall image corresponding to the high-quality portrait is the portrait shooting mode, then the image quality clarity scoring result of the high-quality portrait is the scoring result of the image quality clarity of the image corresponding to the portrait area (i.e., the local clarity scoring result). If it is not a portrait shooting mode, then the scoring area is the entire image. That is, if the shooting mode of the overall image corresponding to the high-quality portrait is not the portrait shooting mode, the image quality clarity scoring result of the high-quality portrait is the scoring result of the global image quality clarity of the overall image corresponding to the high-quality portrait.
可理解的，由于人像模式下人物图像中的背景图像默认均会有一定的模糊程度，则在计算该人物图像的模糊度时，不考虑该人物图像的背景图像的模糊度，得到的画质清晰度评分结果更具代表性。It is understandable that since the background image in the character image in portrait mode will have a certain degree of blur by default, the blur of the background image of the character image is not taken into account when calculating the blur of the character image, and the obtained image quality clarity score result is more representative.
在一种可能的实现方式中，所述第一肖像集合中的任意一个肖像图像的画质清晰度值包括N种清晰度算法对应的N个清晰度值，所述预设画质清晰度阈值包括N个清晰度阈值，所述N个清晰度阈值分别与所述N种清晰度算法对应，图像的所述N个清晰度值分别对应地满足所述N个清晰度阈值表示图像的清晰度好，所述N为大于或等于2的整数，所述第一肖像集合中的任意一个肖像图像的画质清晰度值满足预设画质清晰度阈值包括：所述第一肖像集合中的任意一个肖像图像的所述N个清晰度值分别对应地满足所述N个清晰度阈值。In a possible implementation, the image quality clarity value of any portrait image in the first portrait set includes N clarity values corresponding to N clarity algorithms, the preset image quality clarity threshold includes N clarity thresholds, the N clarity thresholds respectively correspond to the N clarity algorithms, the N clarity values of the image respectively correspondingly satisfy the N clarity thresholds, indicating that the image clarity is good, N is an integer greater than or equal to 2, and the image quality clarity value of any portrait image in the first portrait set satisfies the preset image quality clarity threshold including: the N clarity values of any portrait image in the first portrait set respectively correspondingly satisfy the N clarity thresholds.
示例性的，所述N种清晰度算法包括Brenner梯度函数、Laplacian梯度函数、灰度方差SMD函数、无参考图像评价指标NIQE、Brisque算法中的一项或一项以上。Exemplarily, the N types of clarity algorithms include one or more of a Brenner gradient function, a Laplacian gradient function, a grayscale variance SMD function, a no-reference image evaluation index NIQE, and a Brisque algorithm.
可理解的，相比于仅采用一种清晰度计算方法评估图像的清晰度，采用多种清晰度计算方法评估图像的清晰度得到的评估结果更为客观，基于此，可以综合清晰度的各种不同统计方法的计算结果评估图像的清晰度，进一步提高清晰度评估结果(例如画质评分结果)的客观性。It can be understood that compared with using only one clarity calculation method to evaluate the clarity of an image, the evaluation results obtained by using multiple clarity calculation methods to evaluate the clarity of an image are more objective. Based on this, the calculation results of various different statistical methods of clarity can be combined to evaluate the clarity of the image, thereby further improving the objectivity of the clarity evaluation results (such as image quality scoring results).
在一种可能的实现方式中，所述优质肖像的肖像综合评分结果与所述优质肖像对应的面部评分结果和五官评分结果相关，其中，所述优质肖像对应的面部评分结果与所述优质肖像的面部皮肤纹理相关；所述五官评分结果包括所述优质肖像的眼睛区域评分结果、鼻子区域评分结果、嘴巴区域评分结果、眉毛区域评分结果、以及耳朵区域评分结果，所述眼睛区域评分结果与所述优质肖像中眼睛区域图像的清晰度以及眼睛睁开程度相关，所述鼻子区域评分结果与所述优质肖像中鼻子区域图像的清晰度和高光信息相关，所述嘴巴区域评分结果与所述优质肖像中嘴巴区域图像的清晰度和嘴巴张开程度相关，所述眉毛区域评分结果与所述优质肖像中眉毛区域图像的清晰度相关，所述耳朵区域评分结果与所述优质肖像中耳朵区域图像的清晰度相关。In a possible implementation, the comprehensive portrait scoring result of the high-quality portrait is related to the facial scoring result and the facial features scoring result corresponding to the high-quality portrait, wherein the facial scoring result corresponding to the high-quality portrait is related to the facial skin texture of the high-quality portrait; the facial features scoring results include the eye area scoring result, the nose area scoring result, the mouth area scoring result, the eyebrow area scoring result, and the ear area scoring result of the high-quality portrait, the eye area scoring result is related to the clarity of the eye area image and the degree of eye openness in the high-quality portrait, the nose area scoring result is related to the clarity and highlight information of the nose area image in the high-quality portrait, the mouth area scoring result is related to the clarity of the mouth area image and the degree of mouth openness in the high-quality portrait, the eyebrow area scoring result is related to the clarity of the eyebrow area image in the high-quality portrait, and the ear area scoring result is related to the clarity of the ear area image in the high-quality portrait.
由此，综合各个维度的特征选取与综合图像质量较高的该优质肖像对待增强肖像的肖像整体进行图像增强，提升当前待增强人物图像的细节、质感以及纹理，改善处理后的人物图像中人物的特点可能会包含不属于当前人物的特点的问题。Therefore, the overall image enhancement of the portrait to be enhanced is performed by comprehensively selecting features from various dimensions and the high-quality portrait with high comprehensive image quality, thereby improving the details, texture and texture of the current image of the person to be enhanced, and improving the problem that the characteristics of the person in the processed image may include characteristics that do not belong to the current person.
在一种可能的实现方式中，所述优质图像为属于肖像整体图像的优质肖像，所述利用所述优质图像对所述第一图像中的肖像图像的整体图像和/或局部图像进行图像增强处理，包括：在所述优质肖像的头部姿态与所述第一图像的肖像图像的头部姿态的差值大于或等于第三预设差值的情况下，采用三维人脸重建或NerfGAN技术，基于所述第一图像的肖像图像的头部姿态对所述优质肖像的头部姿态进行矫正，得到第三参考肖像，所述第三参考肖像中的头部姿态与所述第一图像的肖像图像的头部姿态的差值小于第四预设差值；利用所述第三参考肖像对所述第一图像的肖像图像的整体图像进行图像增强处理。In a possible implementation, the high-quality image is a high-quality portrait belonging to the overall image of the portrait, and the use of the high-quality image to perform image enhancement processing on the overall image and/or local image of the portrait image in the first image includes: when the difference between the head posture of the high-quality portrait and the head posture of the portrait image of the first image is greater than or equal to a third preset difference, using three-dimensional face reconstruction or NerfGAN technology to correct the head posture of the high-quality portrait based on the head posture of the portrait image of the first image to obtain a third reference portrait, and the difference between the head posture in the third reference portrait and the head posture of the portrait image of the first image is less than a fourth preset difference; and using the third reference portrait to perform image enhancement processing on the overall image of the portrait image of the first image.
示例性的，若基于上述方式1获取到的优质肖像的头部姿态与第一图像的头部姿态的差值大于或等于第三预设差值，则基于上述方式2获取优质肖像，若基于上述方式2无法获取到优质肖像，则采用方式1获取到的优质肖像以及三维人脸重建或NerfGAN技术，对方式1获取到的优质肖像的头部姿态进行矫正，得到上述第三参考肖像，基于上述第三参考肖像对第一图像的肖像图像进行增强处理。Exemplarily, if the difference between the head posture of the high-quality portrait obtained based on the above-mentioned method 1 and the head posture of the first image is greater than or equal to the third preset difference, the high-quality portrait is obtained based on the above-mentioned method 2; if the high-quality portrait cannot be obtained based on the above-mentioned method 2, the high-quality portrait obtained by method 1 and three-dimensional face reconstruction or NerfGAN technology are used to correct the head posture of the high-quality portrait obtained by method 1 to obtain the above-mentioned third reference portrait, and the portrait image of the first image is enhanced based on the above-mentioned third reference portrait.
例如上述第三预设差值可以为10或20，上述第四预设差值可以为10或20，具体根据需求而定，本文对此不做限定。For example, the third preset difference may be 10 or 20, and the fourth preset difference may be 10 or 20, depending on specific needs, which is not limited in this document.
示例性的，采用三维人脸重建技术基于所述第一图像的肖像图像的头部姿态对所述优质肖像的头部姿态进行矫正，得到第三参考肖像，具体包括：对所述优质肖像进行三维人脸重建，得到三维人脸模型；基于所述第一图像中的肖像图像的头部姿态对所述三维人脸模型进行头部姿态矫正，获取矫正后的三维人脸模型；将所述矫正后的三维人脸模型渲染为二维图像，并将所述矫正后的三维人脸模型作为上述第三参考肖像，利用该二维图像对第一图像的肖像图像进行增强处理。Exemplarily, a three-dimensional face reconstruction technology is used to correct the head posture of the high-quality portrait based on the head posture of the portrait image of the first image to obtain a third reference portrait, specifically including: performing three-dimensional face reconstruction on the high-quality portrait to obtain a three-dimensional face model; performing head posture correction on the three-dimensional face model based on the head posture of the portrait image in the first image to obtain a corrected three-dimensional face model; rendering the corrected three-dimensional face model into a two-dimensional image, and using the corrected three-dimensional face model as the above-mentioned third reference portrait, and using the two-dimensional image to enhance the portrait image of the first image.
在本申请实施例中，当优质肖像的头部姿态与待增强肖像的头部姿态欧拉角的差值大于或等于预设欧拉角阈值时，对优质肖像进行三维人脸重建，可以得到该优质肖像更为完整的脸部信息，且三维人脸重建基于图像质量较好的该优质肖像包含的人脸进行三维人脸重建，得到的三维人脸模型不仅包含更为完整的脸部信息，其图像质量也较好。另外，基于所述待增强肖像的头部姿态对三维人脸模型进行头部姿态矫正，将所述矫正后的三维人脸模型渲染为二维图像，可以获取到三维人脸模型中与待增强肖像头部姿态相同或接近的人脸图像。In an embodiment of the present application, when the difference between the Euler angles of the head posture of the high-quality portrait and the head posture of the portrait to be enhanced is greater than or equal to the preset Euler angle threshold, the high-quality portrait is subjected to three-dimensional face reconstruction, and more complete facial information of the high-quality portrait can be obtained, and the three-dimensional face reconstruction is performed based on the face included in the high-quality portrait with good image quality, and the obtained three-dimensional face model not only contains more complete facial information, but also has better image quality. In addition, the head posture of the three-dimensional face model is corrected based on the head posture of the portrait to be enhanced, and the corrected three-dimensional face model is rendered as a two-dimensional image, so that a face image with the same or similar head posture as the portrait to be enhanced in the three-dimensional face model can be obtained.
也就是说，在对待增强肖像进行增强处理之前，通过3D人脸建模进一步获取与待增强肖像更为匹配且图像质量较好的上述二维图像，其中该二维图像来源于与第一图像的头部姿态相同或差异较小的、图像质量完整、以及图像质量较好的头部姿态矫正后的3D人脸模型。That is to say, before enhancing the portrait to be enhanced, the above-mentioned two-dimensional image that is more closely matched with the portrait to be enhanced and has better image quality is further obtained through 3D face modeling, wherein the two-dimensional image is derived from a 3D face model after head posture correction with the same or smaller difference as the first image, complete image quality, and better image quality.
由此，当数据库中搜索到的比较匹配且图像质量较好的优质肖像，但其头部姿态与待增强肖像的头部姿态仍相差较大的情况下，可以基于3D人脸建模和该优质肖像获取更为完整且与待增强肖像的头部姿态更匹配的图像质量较好的上述二维图像对该待增强图像进行增强，从而基于该二维图像提取到的纹理特征所处区域与待增强肖像中的对应区域匹配，在进行特征拟合计算时所使用的特征信息的区域几何信息更为匹配，可以进一步提高图像增强效果。Therefore, when a high-quality portrait with a relatively good match and good image quality is searched in the database, but its head posture is still quite different from the head posture of the portrait to be enhanced, the above-mentioned two-dimensional image with better image quality that is more complete and more matches the head posture of the portrait to be enhanced can be obtained based on 3D face modeling and the high-quality portrait to enhance the image to be enhanced, so that the area where the texture features extracted from the two-dimensional image are located matches the corresponding area in the portrait to be enhanced, and the regional geometric information of the feature information used in the feature fitting calculation is more matched, which can further improve the image enhancement effect.
在一种可能的实现方式中，所述优质图像包括优质面部图像和优质五官图像，所述优质面部图像包含人脸中的面部区域对应的局部图像，所述优质五官图像包含人脸中五官不同部位对应的局部图像，所述优质面部图像的面部纹理清晰度优于所述第一图像的面部纹理清晰度，所述优质五官图像中的五官清晰度、五官立体感、嘴唇纹理、眼神光的光斑清晰度、眼睛睁开程度、嘴巴张开程度中的一项或多项优于所述第一图像的五官图像。In one possible implementation, the high-quality image includes a high-quality facial image and a high-quality facial features image, the high-quality facial image includes a local image corresponding to the facial area in the human face, the high-quality facial features image includes local images corresponding to different parts of the facial features in the human face, the facial texture clarity of the high-quality facial image is better than the facial texture clarity of the first image, and one or more of the facial feature clarity, facial feature three-dimensionality, lip texture, eye light spot clarity, eye openness, and mouth openness in the high-quality facial features image are better than those of the facial features image of the first image.
在一种可能的实现方式中，所述优质图像包括优质面部图像和优质五官图像，所述优质五官图像包括第一嘴巴图像、第一眼睛图像、第一鼻子图像、第一眉毛图像、第一耳朵图像；其中，所述优质面部图像为面部图像集合中面部评分分值最优的图像，所述面部图像集合中的每一项面部图像来源于所述目标人物，所述面部图像集合中的每一项面部图像的评分结果优于所述第一图像的面部图像的评分结果，所述优质面部图像的面部评分结果与面部纹理细腻度相关；所述第一嘴巴图像为嘴巴图像集合中嘴巴评分分值最优的图像，所述嘴巴图像集合中的每一项嘴巴图像均来源于所述目标人物，所述嘴巴图像集合中的每一项嘴巴图像的评分结果优于所述第一图像的嘴巴图像的评分结果，所述第一嘴巴图像的嘴巴评分结果与嘴唇纹理以及嘴巴张开程度相关；所述第一眼睛图像为眼睛图像集合中眼睛评分分值最优的图像，所述眼睛评分集合中的每一项眼睛图像均来源于所述目标人物，所述眼睛图像集合中的每一项眼睛图像的评分结果优于所述第一图像的眼睛图像的评分结果；所述第一眼睛图像的眼睛评分结果与眼睛图像的清晰度以及眼睛睁开程度相关；所述第一鼻子图像为鼻子图像集合中鼻子评分分值最优的图像，所述鼻子图像集合中的每一项鼻子图像来源于所述目标人物，所述鼻子图像集合中的每一项鼻子图像的评分结果优于所述第一图像的鼻子图像的评分结果，所述第一鼻子图像的鼻子评分结果与鼻子图像的清晰度和高光信息相关；所述第一眉毛图像为眉毛图像集合中眉毛评分分值最优的图像，所述眉毛图像集合中的每一项眉毛图像来源于所述目标人物，所述眉毛图像集合中的每一项眉毛图像的评分结果优于所述第一图像的眉毛图像的评分结果，所述第一眉毛图像的眉毛评分结果与眉毛图像的清晰度和色彩对比度相关；所述第一耳朵图像为耳朵图像集合中耳朵评分分值最优的图像，所述耳朵图像集合中的每一项耳朵图像来源于所述目标人物，所述耳朵图像集合中的每一项耳朵图像的评分结果优于所述第一图像的耳朵图像的评分结果，所述第一耳朵图像的耳朵评分结果与耳朵图像的清晰度相关。在一种可能的实现方式中，所述嘴巴图像集合中的每一项嘴巴图像的嘴巴张开程度小于预设张开程度，所述眼睛图像集合中的每一项眼睛图像均包含眼神光(眼神光即为眼球上的白色光斑)以及眼睛睁开程度大于或等于预设睁开程度。In a possible implementation, the high-quality image includes a high-quality facial image and a high-quality facial features image, and the high-quality facial features image includes a first mouth image, a first eye image, a first nose image, a first eyebrow image, and a first ear image; wherein the high-quality facial image is an image with the best facial score in the facial image set, each facial image in the facial image set is derived from the target person, and the scoring result of each facial image in the facial image set is better than the scoring result of the facial image of the first image, and the facial scoring result of the high-quality facial image is related to the fineness of facial texture; the first mouth image is an image with the best mouth score in the mouth image set, each mouth image in the mouth image set is derived from the target person, and the scoring result of each mouth image in the mouth image set is better than the scoring result of the mouth image of the first image, and the mouth scoring result of the first mouth image is related to the lip texture and the degree of mouth opening; the first eye image is an image with the best eye score in the eye image set, each eye image in the eye scoring set is derived from the target person, and the scoring result of each eye image in the eye image set is better than the eye image of the first image. the eye scoring result of the first eye image is related to the clarity of the eye image and the degree of eye opening; the first nose image is the image with the best nose scoring score in the nose image set, each nose image in the nose image set is derived from the target person, and the scoring result of each nose image in the nose image set is better than the scoring result of the nose image of the first image, and the nose scoring result of the first nose image is related to the clarity and highlight information of the nose image; the first eyebrow image is the image with the best eyebrow scoring score in the eyebrow image set, each eyebrow image in the eyebrow image set is derived from the target person, and the scoring result of each eyebrow image in the eyebrow image set is better than the scoring result of the eyebrow image of the first image, and the eyebrow scoring result of the first eyebrow image is related to the clarity and color contrast of the eyebrow image; the first ear image is the image with the best ear scoring score in the ear image set, each ear image in the ear image set is derived from the target person, and the scoring result of each ear image in the ear image set is better than the scoring result of the ear image of the first image, and the ear scoring result of the first ear image is related to the clarity of the ear image. In one possible implementation, the mouth opening degree of each mouth image in the mouth image set is less than a preset opening degree, and each eye image in the eye image set contains eye light (eye light is the white light spot on the eyeball) and the eye opening degree is greater than or equal to the preset opening degree.
由此，优质面部图像和优质五官图像均包含相应的优质特征，为基于优质图像(历史先验图像)对当前第一图像进行图像增强的增强效果提供保障。Therefore, both the high-quality facial image and the high-quality facial feature image contain corresponding high-quality features, which provide a guarantee for the enhancement effect of image enhancement based on the high-quality image (historical prior image) on the current first image.
在一种可能的实现方式中，所述优质图像为属于肖像整体图像的优质肖像，所述利用所述优质图像对所述第一图像中的肖像图像的整体图像和/或局部图像进行图像增强处理，得到与所述第一图像对应的第二图像，包括：利用所述优质肖像对所述第一图像的肖像图像的进行图像增强处理，得到第一增强图像，所述第一增强图像包含第一增强肖像和所述第一图像的原始背景图像，第一增强肖像的图像质量优于所述第一图像的肖像图像的图像质量；在所述第一增强肖像与所述原始背景图像之间满足色彩不均匀条件或满足光照不协调条件的情况下，利用所述第一增强肖像对所述第一图像的原始背景图像进行色彩校正，得到所述第二图像，所述第二图像中的肖像图像与背景图像之间的色差优于所述第一增强肖像与所述原始背景图像之间色差。In a possible implementation, the high-quality image is a high-quality portrait belonging to the overall image of the portrait, and using the high-quality image to perform image enhancement processing on the overall image and/or local image of the portrait image in the first image to obtain a second image corresponding to the first image includes: using the high-quality portrait to perform image enhancement processing on the portrait image of the first image to obtain a first enhanced image, the first enhanced image includes a first enhanced portrait and an original background image of the first image, and the image quality of the first enhanced portrait is better than the image quality of the portrait image of the first image; when the first enhanced portrait and the original background image meet a color unevenness condition or a lighting inharmoniousness condition, using the first enhanced portrait to perform color correction on the original background image of the first image to obtain the second image, and the color difference between the portrait image and the background image in the second image is better than the color difference between the first enhanced portrait and the original background image.
或者也可以理解为，所述基于所述优质图像中的面部图像对所述第一图像中的面部图像的面部纹理进行增强处理，和/或，基于所述优质图像中的五官图像对所述第一图像中对应的五官图像进行增强处理，包括：利用所述优质肖像对所述第一图像的肖像图像的进行图像增强处理，得到第一增强图像，所述第一增强图像包含第一增强肖像和所述第一图像的原始背景图像，第一增强肖像的图像质量优于所述第一图像的肖像图像的图像质量；在所述第一增强肖像与所述原始背景图像之间满足色彩不均匀条件或满足光照不协调条件的情况下，利用所述第一增强肖像对所述第一图像的原始背景图像进行色彩校正，得到所述第二图像，所述第二图像中的肖像图像与背景图像之间的色差优于所述第一增强肖像与所述原始背景图像之间色差。Alternatively, it can also be understood that the facial texture of the facial image in the first image is enhanced based on the facial image in the high-quality image, and/or the corresponding facial features image in the first image is enhanced based on the facial features image in the high-quality image, including: using the high-quality portrait to perform image enhancement processing on the portrait image of the first image to obtain a first enhanced image, the first enhanced image includes a first enhanced portrait and an original background image of the first image, and the image quality of the first enhanced portrait is better than the image quality of the portrait image of the first image; when the first enhanced portrait and the original background image meet the color unevenness condition or the lighting incoordination condition, using the first enhanced portrait to perform color correction on the original background image of the first image to obtain the second image, and the color difference between the portrait image and the background image in the second image is better than the color difference between the first enhanced portrait and the original background image.
可理解的，利用第一增强肖像的光照色彩信息对原始背景图像进行色彩校正，可以提高第一图像的背景图像的清晰度，以及协调处理得到的第二图像的图像整体色差，由此，进一步改善图像的光照质量、以及协调图像整体色差，使得增强得到的第二图像整体更为自然。It can be understood that using the lighting color information of the first enhanced portrait to perform color correction on the original background image can improve the clarity of the background image of the first image and coordinate the overall color difference of the second image obtained through processing, thereby further improving the lighting quality of the image and coordinating the overall color difference of the image, making the enhanced second image more natural as a whole.
在一种可能的实现方式中，所述方法还包括：对所述第一增强肖像中的第一区域进行色彩校正，所述第一区域包括所述第一增强肖像中色彩不均匀的区域、光照不协调的区域、欠曝区域、过曝区域中的一项或一项以上。In a possible implementation, the method further includes: performing color correction on a first area in the first enhanced portrait, the first area including one or more of an area with uneven color, an area with uncoordinated lighting, an underexposed area, and an overexposed area in the first enhanced portrait.
由此，对第一增强肖像中色彩不均匀的区域、光照不协调的区域、欠曝区域、或过曝区域进行色彩校正，可以进一步改善增强后的图像质量，例如可以进一步改善第一增强肖像的图像区域的光照的一致性，以及进一步改善第二图像的真实感。Therefore, by performing color correction on areas with uneven color, areas with inconsistent lighting, underexposed areas, or overexposed areas in the first enhanced portrait, the quality of the enhanced image can be further improved. For example, the consistency of lighting in the image area of the first enhanced portrait can be further improved, and the realism of the second image can be further improved.
在一种可能的实现方式中，所述优质图像包括优质面部图像和优质五官图像，所述利用所述优质图像对所述第一图像中的肖像图像的整体图像和/或局部图像进行图像增强处理，得到与所述第一图像对应的第二图像，包括：利用所述优质五官图像对所述第一图像的五官图像进行五官增强处理，得到增强后的五官图像，以及，利用所述优质面部图像对所述第一图像的面部图像进行面部增强处理，得到增强后的面部图像；基于所述增强后的五官图像、增强后的面部图像、剩余图像、以及所述原始背景图像融合得到第二增强图像，所述第二增强图像包含第二增强肖像和所述第一图像的原始背景图像，所述第二增强肖像的图像质量优于所述第一图像的肖像图像的图像质量；在所述第二增强肖像与所述原始背景图像之间满足色彩不均匀条件或满足光照不协调条件的情况下，利用所述第一增强肖像对所述第一图像的原始背景图像进行色彩校正，得到所述第二图像，所述第二图像中的肖像图像与背景图像之间的色差优于所述第一增强肖像与所述原始背景图像之间的色差。In a possible implementation, the high-quality image includes a high-quality facial image and a high-quality facial features image, and the using of the high-quality image to perform image enhancement processing on the overall image and/or partial image of the portrait image in the first image to obtain a second image corresponding to the first image includes: using the high-quality facial features image to perform facial features enhancement processing on the facial features image of the first image to obtain an enhanced facial features image, and using the high-quality facial image to perform facial enhancement processing on the facial image of the first image to obtain an enhanced facial image; based on the enhanced facial features image, the enhanced facial image, the remaining image, and the original background image, a second enhanced image is obtained by fusing the enhanced facial features image, the enhanced facial image, the remaining image, and the original background image, the second enhanced image includes a second enhanced portrait and the original background image of the first image, and the image quality of the second enhanced portrait is better than the image quality of the portrait image of the first image; when the second enhanced portrait and the original background image meet a color unevenness condition or an illumination incoordination condition, the first enhanced portrait is used to perform color correction on the original background image of the first image to obtain the second image, and the color difference between the portrait image and the background image in the second image is better than the color difference between the first enhanced portrait and the original background image.
上述基于所述增强后的五官图像、增强后的面部图像、剩余图像、以及所述原始背景图像融合得到第二增强图像，具体可以包括：对增强后的五官图像、增强后的面部图像、剩余图像进行拼图处理，以及对图像拼图产生的边缘进行平滑处理，得到第一增强肖像；对第一增强肖像和第一图像的原始背景图像进行拼图处理以及对图像拼图产生的边缘进行平滑处理，得到上述第二增强图像；上述利用所述第一增强肖像对所述第一图像的原始背景图像进行色彩校正，得到所述第二图像具体可以包括：基于色彩迁移技术和该第一增强肖像的色彩信息，对原始背景图像对应区域的色彩信息进行色彩校正，得到所述第二图像。由此，进一步改善图像的光照质量、以及协调图像整体色差，使得增强得到的第二图像整体更为自然。The above-mentioned second enhanced image is obtained by fusing the enhanced facial features image, the enhanced facial image, the remaining image, and the original background image, which may specifically include: performing jigsaw processing on the enhanced facial features image, the enhanced facial image, and the remaining image, and smoothing the edges generated by the image jigsaw to obtain the first enhanced portrait; performing jigsaw processing on the first enhanced portrait and the original background image of the first image, and smoothing the edges generated by the image jigsaw to obtain the above-mentioned second enhanced image; the above-mentioned color correction of the original background image of the first image using the first enhanced portrait to obtain the second image may specifically include: performing color correction on the color information of the corresponding area of the original background image based on the color migration technology and the color information of the first enhanced portrait to obtain the second image. In this way, the lighting quality of the image is further improved, and the overall color difference of the image is coordinated, so that the enhanced second image is more natural as a whole.
可理解的，利用第二增强肖像的光照色彩信息对原始背景图像进行色彩校正，可以提高第一图像的背景图像的清晰度，以及协调处理得到的第二图像的图像整体色差，使得增强得到的第二图像整体更为自然。It is understandable that color correction of the original background image using the illumination color information of the second enhanced portrait can improve the clarity of the background image of the first image and coordinate the overall color difference of the second image obtained by processing, making the enhanced second image more natural as a whole.
在一种可能的实现方式中，所述方法还包括：对所述第二增强肖像中的第二区域进行色彩校正，所述第二区域包括所述第二增强肖像中色彩不均匀的区域、光照不协调的区域、欠曝区域、过曝区域中的一项或一项以上。In a possible implementation, the method further includes: performing color correction on a second area in the second enhanced portrait, the second area including one or more of an area with uneven color, an area with uncoordinated lighting, an underexposed area, and an overexposed area in the second enhanced portrait.
可理解的，第一增强肖像的五官图像基于优质五官图像增强得到，该优质五官图像中的每个五官区域图像可能来源于同一人物的多种不同图像，也就是说优质五官图像中的每个单独的五官区域图像之间的光照、色彩、曝光信息会有所差异，则增强后的各个五官图像之间也可能会存在明显的光照不均匀、色彩不均匀或过曝欠曝问题，基于此，在得到上述增强肖像A后，再对增强肖像A中的色彩不均匀的区域、光照不协调的区域、欠曝区域、过曝区域进行色彩校正，改善增强后的各个五官图像之间也可能会存在明显的光照不均匀、色彩不均匀、过曝、或欠曝的问题。It is understandable that the facial feature image of the first enhanced portrait is obtained based on the enhancement of the high-quality facial feature image. Each facial feature area image in the high-quality facial feature image may come from multiple different images of the same person. That is to say, the lighting, color, and exposure information between each individual facial feature area image in the high-quality facial feature image may be different. Therefore, there may be obvious lighting unevenness, color unevenness, or overexposure or underexposure problems between the enhanced facial feature images. Based on this, after obtaining the above-mentioned enhanced portrait A, color correction is performed on the color uneven areas, lighting inharmonious areas, underexposed areas, and overexposed areas in the enhanced portrait A to improve the obvious lighting unevenness, color unevenness, overexposure, or underexposure problems between the enhanced facial feature images.
在一种可能的实现方式中，所述获取与所述第一图像匹配的优质图像包括：在所述电子设备的设备状态满足预设设备状态的情况下，获取所述优质图像，所述预设设备状态包括所述电子设备的当前电量大于或等于预设电量阈值，或者，所述预设设备状态包括所述电子设备的当前电量大于或等于预设电量阈值且所述电子设备处于充电灭屏状态；或者，在所述电子设备接收到目标处理请求的情况下，获取所述优质图像，所述目标处理请求用于表示用户主动请求对所述第一图像进行图像处理。In one possible implementation, acquiring a high-quality image that matches the first image includes: acquiring the high-quality image when a device state of the electronic device satisfies a preset device state, the preset device state including that a current power level of the electronic device is greater than or equal to a preset power level threshold, or the preset device state including that a current power level of the electronic device is greater than or equal to a preset power level threshold and the electronic device is in a charging and screen-off state; or acquiring the high-quality image when the electronic device receives a target processing request, the target processing request being used to indicate that a user actively requests image processing of the first image.
在本申请实施例中，若用户主动请求对第一图像进行图像处理，则立即获取优质图像第一图像进行增强处理，优先满足用户需求。若用户未主动请求对第一图像进行图像处理，则电子设备可以在设备满足当前电量大于或等于预设电量阈值，或者，当前电量大于或等于预设电量阈值且设备处于充电灭屏状态的情况下，获取优质图像对第一图像进行图像增强，改善图像增强对设备的电能损耗问题，由此，在满足用户需求的同时改善电能损耗问题。In the embodiment of the present application, if the user actively requests to process the first image, a high-quality image is immediately obtained for enhancement processing of the first image, giving priority to satisfying the user's needs. If the user does not actively request to process the first image, the electronic device can obtain a high-quality image to enhance the first image when the current power of the device is greater than or equal to the preset power threshold, or when the current power is greater than or equal to the preset power threshold and the device is in a charging and screen-off state, thereby improving the problem of power consumption of the device caused by image enhancement, thereby satisfying the user's needs while improving the problem of power consumption.
在一种可能的实现方式中，所述获取第一图像包括：基于相机应用拍摄获取所述第一图像；所述在所述电子设备的设备状态满足预设设备状态的情况下，获取所述优质图像，包括：在所述电子设备的当前电量大于或等于预设电量阈值的情况下，获取所述优质图像；或者，在所述电子设备的当前电量小于预设电量阈值的情况下，存储所述第一图像，并在所述电子设备再次满足当前电量大于或等于所述预设电量阈值且处于充电灭屏状态的条件后，或在所述电子设备接收到用户主动触发的关于所述第一图像的图像处理请求后，获取所述优质图像。In one possible implementation, acquiring the first image includes: acquiring the first image based on shooting with a camera application; acquiring the high-quality image when the device state of the electronic device satisfies a preset device state, including: acquiring the high-quality image when the current power of the electronic device is greater than or equal to a preset power threshold; or, storing the first image when the current power of the electronic device is less than a preset power threshold, and acquiring the high-quality image after the electronic device again satisfies the condition that the current power is greater than or equal to the preset power threshold and is in a charging screen-off state, or after the electronic device receives an image processing request for the first image actively triggered by the user.
示例性的，若用户当前通过电子设备在户外拍摄M(M大于或等于1)张人物图像，电子设备在确定当前电量大于或等于预设电量阈值时，采用本申请提供的任一种图像处理方法对该M张人物图像进行图像增强，当处理到第X张(X小于或等于M)人物图像后发现电子设备的当前电量小于预设电量阈值时，则停止对其他人物图像进行增强处理，从而可以为电子设备保持相当的电量以保障用户的户外活动的设备需求。Exemplarily, if a user currently uses an electronic device to take M (M is greater than or equal to 1) images of people outdoors, when the electronic device determines that the current power level is greater than or equal to a preset power threshold, it uses any image processing method provided in the present application to enhance the M images of people. When after processing the Xth (X is less than or equal to M) image of a person and finding that the current power level of the electronic device is less than the preset power threshold, it stops enhancing other images of the person, thereby maintaining a sufficient power level for the electronic device to ensure the equipment needs of the user's outdoor activities.
在一种可能的实现方式中，也可以向用户提供是否启用本申请提供的图像处理方法的功能控件，默认启用本申请提供的图像处理方法，但也可以基于用户针对该功能控件的关闭或开启操作相应地关闭或启用本申请提供的图像处理方法。In one possible implementation, a function control for determining whether to enable the image processing method provided in the present application may be provided to the user. The image processing method provided in the present application is enabled by default, but the image processing method provided in the present application may also be turned off or on based on the user's turning off or on operation of the function control.
在一种可能的实现方式中，所述获取第一图像包括：基于数据下载方式在第一时间内获取至少两张以上所述第一图像；所述在所述电子设备的设备状态满足预设设备状态的情况下，获取所述优质图像，包括：在所述电子设备的当前电量大于或等于所述预设电量阈值且处于充电灭屏状态的情况下，获取所述优质图像；或者，在所述电子设备接收到用户主动触发的关于所述第一图像的图像处理请求的情况下，获取所述优质图像。In a possible implementation, acquiring the first image includes: acquiring at least two or more of the first images within a first time based on a data download method; acquiring the high-quality image when the device state of the electronic device satisfies a preset device state, including: acquiring the high-quality image when the current power of the electronic device is greater than or equal to the preset power threshold and is in a charging screen-off state; or acquiring the high-quality image when the electronic device receives an image processing request for the first image actively triggered by a user.
可理解的，数据传输例如蓝牙、NFC传输等就有一定的电能损耗，且若用户通过蓝牙或NFC等方式获取第一图像，也可以理解为当前用户对设备的使用需求较高，则基于本申请提供的方法，这种场景下电子设备需要满足电量大于或等于所述预设电量阈值且处于充电灭屏状态的情况，才能对获取到的人物图像进行增强处理，在达到图像增强效果的同时，改善图像增强为设备带来的电能损耗问题，以及为电子设备保持较好的电量状态以保障用户的设备使用需求。It is understandable that data transmission, such as Bluetooth, NFC transmission, etc., will have a certain amount of power consumption, and if the user obtains the first image through Bluetooth or NFC, etc., it can also be understood that the current user has a high demand for the use of the device. Based on the method provided in the present application, in this scenario, the electronic device needs to meet the condition that the power is greater than or equal to the preset power threshold and is in a charging and screen-off state before it can enhance the acquired character image. While achieving the image enhancement effect, it improves the power loss problem caused by image enhancement to the device, and maintains a good power state for the electronic device to ensure the user's device usage needs.
在一种可能的实现方式中，所述获取与所述第一图像匹配的优质图像包括：在确定所述第一图像满足预设增强条件的情况下，将所述历史人物图像作为所述第一图像，所述预设增强条件包括以下一项或一项以上：人像模糊、人脸纹理细节缺失、人脸区域图像噪声高、人脸区域光照不足、人脸区域光照过亮。In one possible implementation, obtaining a high-quality image that matches the first image includes: when it is determined that the first image meets preset enhancement conditions, using the image of the historical figure as the first image, the preset enhancement conditions include one or more of the following: blurred portrait, missing facial texture details, high image noise in the face area, insufficient lighting in the face area, and excessive lighting in the face area.
在一种可能的实现方式中，所述方法应用于电子设备，所述电子设备中存储有包含肖像图像的历史人物图像，所述历史人物图像基于数据下载方式或相机拍照方式获得，所述方法还包括：在确定所述电子设备当前电量大于或等于预设电量阈值且所述电子设备处于充电灭屏状态、以及所述历史任务图像满足预设增强条件的情况下，将所述历史人物图像作为所述第一图像，所述预设增强条件包括以下一项或一项以上：人像模糊、人脸纹理细节缺失、人脸区域图像噪声高、人脸区域光照不足、人脸区域光照过亮。In one possible implementation, the method is applied to an electronic device, which stores an image of a historical figure including a portrait image, and the image of the historical figure is obtained based on a data download method or a camera photo method. The method also includes: when it is determined that the current power of the electronic device is greater than or equal to a preset power threshold and the electronic device is in a charging and screen-off state, and the historical task image meets a preset enhancement condition, using the image of the historical figure as the first image, and the preset enhancement condition includes one or more of the following: blurred portrait, missing facial texture details, high image noise in the facial area, insufficient lighting in the facial area, and excessive lighting in the facial area.
第二方面，本申请实施例提供一种图像处理系统，所述图像处理系统包括增强子系统和人物特征检索子系统，所述增强子系统包含融合增强模块，其中，所述人物特征检索子系统，用于在获取第一图像之后，获取与所述第一图像匹配的优质图像；所述融合增强模块，用于利用所述优质图像对所述第一图像中的肖像图像的整体图像和/或局部图像进行图像增强处理，得到第二图像。In a second aspect, an embodiment of the present application provides an image processing system, which includes an enhancement subsystem and a character feature retrieval subsystem, wherein the enhancement subsystem includes a fusion enhancement module, wherein the character feature retrieval subsystem is used to obtain a high-quality image matching the first image after obtaining the first image; and the fusion enhancement module is used to use the high-quality image to perform image enhancement processing on the overall image and/or local image of the portrait image in the first image to obtain a second image.
在一种可能的实现方式中，所述融合增强模块，具体用于利用所述优质肖像对所述待增强肖像的整体肖像进行图像增强处理，得到所述第一增强图像。In a possible implementation, the fusion enhancement module is specifically configured to use the high-quality portrait to perform image enhancement processing on the overall portrait of the portrait to be enhanced, so as to obtain the first enhanced image.
在一种可能的实现方式中，所述图像处理系统还包括先验知识管理子系统、图像数据库以及人物处理子系统，所述人物处理子系统包括人像处理模块、评分模块以及人脸属性估计模块，所述先验知识管理子系统用于为人物图像创建索引，所述图像数据库用于存储与人物图像(包括所述优质图像)相关的以下数据：索引、该优质图像对应的原始图像资源、人像抠图数据、人脸解析数据、人脸聚类标签、头部姿态欧拉角、画质评分结果、五官评分结果、面部评分结果。所述人像处理模块用于对人物图像进行人脸聚类获取对应的人脸聚类标签；所述人像处理模块还用于对所述人物图像进行人像抠图和人脸解析以获取对应的人像抠图数据和人脸解析数据，所述评分模块用于基于人物图像的人像抠图数据和人脸解析数据对人物图像进行面部评分和五官评分，以获取对应的面部评分结果和五官评分结果，所述评分模块还用于对人物图像进行画质评分，以获取对应的画质评分结果。所述人脸属性估计模块用于对人物图像进行姿态估计得到对应的头部姿态欧拉角。In a possible implementation, the image processing system further includes a priori knowledge management subsystem, an image database, and a character processing subsystem, the character processing subsystem includes a portrait processing module, a scoring module, and a face attribute estimation module, the priori knowledge management subsystem is used to create an index for the character image, and the image database is used to store the following data related to the character image (including the high-quality image): index, original image resources corresponding to the high-quality image, portrait cutout data, face analysis data, face clustering labels, head posture Euler angles, image quality scoring results, facial features scoring results, and facial scoring results. The portrait processing module is used to perform face clustering on the character image to obtain the corresponding facial clustering labels; the portrait processing module is also used to perform portrait cutout and face analysis on the character image to obtain the corresponding portrait cutout data and face analysis data, the scoring module is used to perform facial scoring and facial features scoring on the character image based on the portrait cutout data and face analysis data of the character image to obtain the corresponding facial scoring results and facial features scoring results, and the scoring module is also used to perform image quality scoring on the character image to obtain the corresponding image quality scoring results. The face attribute estimation module is used to estimate the posture of the character image to obtain the corresponding head posture Euler angle.
在一种可能的实现方式中，所述人物处理子系统还包括3D重建模块，所述3D重建模块，用于在所述优质肖像的头部姿态欧拉角与所述待增强肖像的头部姿态欧拉角的差值大于或等于预设欧拉角阈值的情况下，对所述优质肖像进行三维人脸重建，得到三维人脸模型；基于所述待增强肖像的头部姿态对所述三维人脸模型进行头部姿态矫正，获取矫正后的三维人脸模型；以及将所述矫正后的三维人脸模型渲染为二维图像，得到与所述优质肖像对应的头部姿态矫正后的二维图像；所述融合增强模块，具体用于利用所述二维图像对所述待增强肖像进行增强处理，得到所述第一增强图像。In a possible implementation, the character processing subsystem also includes a 3D reconstruction module, which is used to perform three-dimensional face reconstruction on the high-quality portrait to obtain a three-dimensional face model when the difference between the Euler angle of the head posture of the high-quality portrait and the Euler angle of the head posture of the portrait to be enhanced is greater than or equal to a preset Euler angle threshold; perform head posture correction on the three-dimensional face model based on the head posture of the portrait to be enhanced to obtain a corrected three-dimensional face model; and render the corrected three-dimensional face model into a two-dimensional image to obtain a two-dimensional image with head posture correction corresponding to the high-quality portrait; the fusion enhancement module is specifically used to use the two-dimensional image to perform enhancement processing on the portrait to be enhanced to obtain the first enhanced image.
在一种可能的实现方式中，所述融合增强模块，还用于对所述第一图像中包含的眼神光的光斑轮廓清晰度和光斑亮度进行增强处理；获取虹膜信息库与所述第一图像的虹膜色彩信息的相似度较高的第一参考虹膜图像，基于所述第一参考虹膜图像对所述第一图像的眼睛部位的虹膜进行虹膜纹理和虹膜色彩的增强处理；对所述第一图像中的眉毛区域的图像进行对比度增强处理；对所述第一图像中的嘴唇区域的图像进行色彩和纹理增强处理。In a possible implementation, the fusion enhancement module is further used to enhance the spot contour clarity and spot brightness of the eye light contained in the first image; obtain a first reference iris image with a high degree of similarity between the iris information library and the iris color information of the first image, and enhance the iris texture and iris color of the iris in the eye area of the first image based on the first reference iris image; perform contrast enhancement on the image of the eyebrow area in the first image; and perform color and texture enhancement on the image of the lip area in the first image.
在一种可能的实现方式中，所述融合增强模块，还用于在增强后的第一增强肖像与原始背景图像之间满足色彩不均匀条件或满足光照不协调条件的情况下，利用第一增强肖像对第一图像的原始背景图像进行色彩校正，得到第二图像，第二图像中的肖像图像与背景图像之间的色差优于第一增强肖像与原始背景图像之间色差。In a possible implementation, the fusion enhancement module is further used to use the first enhanced portrait to perform color correction on the original background image of the first image when the enhanced first enhanced portrait and the original background image meet a color unevenness condition or a lighting inharmoniousness condition, so as to obtain a second image, in which the color difference between the portrait image and the background image is better than the color difference between the first enhanced portrait and the original background image.
在一种可能的实现方式中，所述融合增强模块，还用于对第一增强肖像中的第一区域进行色彩校正，该第一区域包括第一增强肖像中色彩不均匀的区域、光照不协调的区域、欠曝区域、过曝区域中的一项或一项以上。In a possible implementation, the fusion enhancement module is further used to perform color correction on a first area in the first enhanced portrait, where the first area includes one or more of an area with uneven color, an area with uncoordinated lighting, an underexposed area, and an overexposed area in the first enhanced portrait.
在一种可能的实现方式中，所述融合增强模块，具体用于利用所述优质五官图像对所述待增强五官图像进行五官增强处理，得到增强后的五官图像，以及，利用所述优质面部图像对所述待增强面部图像进行面部增强处理，得到增强后的面部图像；对所述增强后的五官图像、所述增强后的面部图像、以及剩余图像进行融合处理，得到所述第一增强图像，所述剩余图像为所述第一图像的肖像中除了所述待增强面部图像和所述待增强五官图像之外的其他图像。In a possible implementation, the fusion enhancement module is specifically used to use the high-quality facial features image to perform facial features enhancement processing on the facial features image to be enhanced to obtain an enhanced facial features image, and to use the high-quality facial image to perform facial enhancement processing on the facial image to be enhanced to obtain an enhanced facial image; and to perform fusion processing on the enhanced facial features image, the enhanced facial image, and the remaining image to obtain the first enhanced image, wherein the remaining image is other images in the portrait of the first image except the facial image to be enhanced and the facial features image to be enhanced.
在一种可能的实现方式中，所述融合增强模块，具体用于对所述第一增强图像和所述原始背景图像进行拼图处理，以及对所述第一增强肖像和所述原始背景图像拼图产生的边缘进行平滑处理，得到拼图以及平滑处理后的中间图像；基于色彩迁移技术和所述中间图像中所述第一增强图像对应区域的色彩信息，对所述中间图像中所述原始背景图像对应区域的色彩信息进行色彩校正，得到所述第二图像。In a possible implementation, the fusion enhancement module is specifically used to perform puzzle processing on the first enhanced image and the original background image, and to smooth the edges generated by the puzzle of the first enhanced image and the original background image, so as to obtain an intermediate image after the puzzle and smoothing; based on the color migration technology and the color information of the area corresponding to the first enhanced image in the intermediate image, color correction is performed on the color information of the area corresponding to the original background image in the intermediate image to obtain the second image.
在一种可能的实现方式中，所述人物处理子系统还包括特征提取模块，所述增强子系统还包括判断模块，所述特征提取模块中的人脸检测模块用于对第一图像进行人脸检测，所述判断模块用于确定第一图像是否满足预设增强条件，所述人物特征检索子系统，具体用于在所述第一图像中包含人脸图像、且所述第一图像满足预设增强条件中的至少一项的情况下，获取所述优质图像。In a possible implementation, the character processing subsystem also includes a feature extraction module, and the enhancement subsystem also includes a judgment module. The face detection module in the feature extraction module is used to perform face detection on the first image, and the judgment module is used to determine whether the first image meets the preset enhancement conditions. The character feature retrieval subsystem is specifically used to obtain the high-quality image when the first image contains a face image and the first image meets at least one of the preset enhancement conditions.
关于优质图像、优质肖像、优质肖像的头部姿态欧拉角、优质面部图像、优质五官图像、人像抠图数据、人脸解析数据、头部姿态欧拉角、画质评分结果、五官评分结果、面部评分结果等名词的说明以及优质肖像、优质面部图像、优质五官图像的获取方法，可以参照第一方面或第一方面的任意可能的实现方式所示的方法中的相关说明，在此不再详述。Regarding the explanations of terms such as high-quality images, high-quality portraits, Euler angles of head posture of high-quality portraits, high-quality facial images, high-quality facial feature images, portrait cutout data, face analysis data, Euler angles of head posture, image quality scoring results, facial feature scoring results, facial scoring results, and methods for obtaining high-quality portraits, high-quality facial images, and high-quality facial feature images, please refer to the relevant instructions in the method shown in the first aspect or any possible implementation of the first aspect, and will not be described in detail here.
第三方面，本申请实施例提供一种电子设备，所述电子设备包括：一个或多个处理器、存储器和显示屏；所述存储器与所述一个或多个处理器耦合，所述存储器用于存储计算机程序代码，所述计算机程序代码包括计算机指令，所述一个或多个处理器调用所述计算机指令以使得所述电子设备执行所述第一方面或第一方面的任意可能的实现方式所示的方法。In a third aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors, a memory and a display screen; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the method shown in the first aspect or any possible implementation of the first aspect.
第四方面，本申请实施例提供一种芯片系统，所述芯片系统应用于电子设备，所述芯片系统包括一个或多个处理器，所述处理器用于调用计算机指令以使得所述电子设备执行所述第一方面或第一方面的任意可能的实现方式所示的方法。In a fourth aspect, an embodiment of the present application provides a chip system, which is applied to an electronic device, and the chip system includes one or more processors, and the processor is used to call computer instructions so that the electronic device executes the method shown in the first aspect or any possible implementation of the first aspect.
第五方面，本申请实施例提供一种包含指令的计算机程序产品，当所述计算机程序产品在电子设备上运行时，使得所述电子设备执行所述第一方面或第一方面的任意可能的实现方式所示的方法。In a fifth aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed on an electronic device, enables the electronic device to execute the method shown in the first aspect or any possible implementation of the first aspect.
第六方面，本申请实施例提供一种计算机可读存储介质，包括指令，当所述指令在电子设备上运行时，使得所述电子设备执行所述第一方面或第一方面的任意可能的实现方式所示的方法。In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, comprising instructions, which, when executed on an electronic device, enable the electronic device to execute the method shown in the first aspect or any possible implementation of the first aspect.
可以理解的，上述第二方面提供的图像增强系统、第三方面提供的电子设备、第四方面提供的芯片、第五方面提供的计算机程序产品、以及第六方面提供的计算机存储介质均用于执行本申请实施例所提供的方法。因此，其所能达到的有益效果可参考对应方法中的有益效果，此处不再赘述。It can be understood that the image enhancement system provided in the second aspect, the electronic device provided in the third aspect, the chip provided in the fourth aspect, the computer program product provided in the fifth aspect, and the computer storage medium provided in the sixth aspect are all used to execute the method provided in the embodiment of the present application. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method, which will not be repeated here.
附图说明BRIEF DESCRIPTION OF THE DRAWINGS
图1为本申请实施例提供的一种对人物图像进行人像抠图、人脸解析的示意图；FIG1 is a schematic diagram of performing portrait cutout and face analysis on a human image provided by an embodiment of the present application;
图2为本申请实施例提供的一种对人物图像进行图像处理的系统图；FIG2 is a system diagram for performing image processing on a person image provided by an embodiment of the present application;
图3A至图3C为本申请实施例提供的一种系统图中包含的各个子系统中包含的功能模块的示意图；3A to 3C are schematic diagrams of functional modules included in each subsystem included in a system diagram provided in an embodiment of the present application;
图4为本申请实施例提供的一种图像增强方法的流程示意图；FIG4 is a schematic diagram of a flow chart of an image enhancement method provided in an embodiment of the present application;
图5为本申请实施例提供的又一种图像增强方法的流程示意图；FIG5 is a schematic diagram of a flow chart of another image enhancement method provided in an embodiment of the present application;
图6为本申请实施例提供的又一种图像增强方法的流程示意图；FIG6 is a schematic diagram of a flow chart of another image enhancement method provided in an embodiment of the present application;
图7为本申请实施例提供的又一种图像增强方法的流程示意图；FIG7 is a schematic diagram of a flow chart of another image enhancement method provided in an embodiment of the present application;
图8为本申请实施例提供的又一种图像增强方法的流程示意图；FIG8 is a schematic diagram of a flow chart of another image enhancement method provided in an embodiment of the present application;
图9为本申请实施例提供的又一种图像增强方法的时序图；FIG9 is a timing diagram of another image enhancement method provided in an embodiment of the present application;
图10为本申请实施例提供的一种拍照场景下图像增强的用户界面示意图；FIG10 is a schematic diagram of a user interface for image enhancement in a photo-taking scenario provided by an embodiment of the present application;
图11为本申请实施例提供的一种用户主动触发增强请求的场景下与图像增强相关的用户界面示意图；FIG11 is a schematic diagram of a user interface related to image enhancement in a scenario where a user actively triggers an enhancement request provided by an embodiment of the present application;
图12为本申请实施例提供的电子设备100的结构示意图；FIG12 is a schematic diagram of the structure of an electronic device 100 provided in an embodiment of the present application;
图13是本申请实施例的电子设备100的软件结构框图。FIG. 13 is a software structure block diagram of the electronic device 100 according to an embodiment of the present application.
具体实施方式Detailed ways
为了使本申请的目的、技术方案和优点更加清楚，下面将结合附图对本申请作进一不地描述。In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described below in conjunction with the accompanying drawings.
本申请的说明书、权利要求书及附图中的术语“第一”和“第二”等仅用于区别不同对象，而不是用于描述特定顺序。此外，术语“包括”和“具有”以及它们的任何变形，意图在于覆盖不排他的包含。例如包含了一系列步骤或单元的过程、方法、系统、产品或设备等，没有限定于已列出的步骤或单元，而是可选地还包括没有列出的步骤或单元等，或可选地还包括对于这些过程、方法、产品或设备等固有的其它步骤或单元。The terms "first" and "second" in the specification, claims and drawings of this application are only used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.
在本文中提及的“实施例”意味着，结合实施例描述的特定特征、结构或特性可以包含在本申请的至少一个实施例中。在说明书中的各个位置出现该短语并不一定均是指相同的实施例，也不是与其它实施例互斥的独立的或备选的实施例。本领域技术人员可以显式地和隐式地理解的是，本文所描述的实施例可以与其它实施例相结合。The "embodiment" mentioned in this article means that the specific features, structures or characteristics described in conjunction with the embodiment can be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It can be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.
在本申请中，“至少一个(项)”是指一个或者多个，“多个”是指两个或两个以上，“至少两个(项)”是指两个或三个及三个以上，“和/或”，用于描述关联对象的关联关系，表示可以存在三种关系，例如，“A和/或B”可以表示：只存在A，只存在B以及同时存在A和B三种情况，其中A，B可以是单数或者复数。字符“/”一般表示前后关联对象是一种“或”的关系。“以下至少一项(个)”或其类似表达，是指这些项中的任意组合。例如，a，b或c中的至少一项(个)，可以表示：a，b，c，“a和b”，“a和c”，“b和c”，或“a和b和c”。In the present application, "at least one (item)" means one or more, "more than one" means two or more, "at least two (items)" means two or three and more than three, and "and/or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and/or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character "/" generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c".
为了便于理解，以下示例地给出了部分与本申请实施例中相关概念的说明以供参考。To facilitate understanding, some of the concepts related to the embodiments of the present application are described below by way of example for reference.
(1)肖像、背景图像、人脸图像、五官图像、面部图像(1) Portrait, background image, face image, facial features image, facial image
在本申请中，电子设备可以对包含人脸的图像进行图像增强处理。以下为便于描述，将包含人脸的图像称为人物图像，该人像图像可以包含背景图像和肖像，或者，该人物图像也可以仅包含肖像，或者该人物图像也可以仅包含人脸图像，本文对此不做限定。In the present application, an electronic device can perform image enhancement processing on an image containing a human face. For ease of description, an image containing a human face is referred to as a person image. The person image can include a background image and a portrait, or the person image can include only a portrait, or the person image can include only a face image, which is not limited in this document.
其中，肖像是指人物图像中能辨别出人物身份信息的人体区域对应的图像，肖像可以是包含人脸图像的人物全身照或半身照，或者肖像也可以是仅包含人脸的照片，本文对此不做限定。Among them, a portrait refers to an image corresponding to the human body area in a person image that can identify the person's identity information. The portrait can be a full-body or half-body photo of a person including a face image, or a portrait can be a photo containing only the face. This article does not limit this.
背景图像是指图像中除了肖像区域之外的其他区域的图像。The background image refers to the image of other areas except the portrait area in the image.
人脸图像为人物图像中人物的脸部区域对应的图像，人脸图像包含的部位可以划分为：五官、除五官之外的其他连续的面部皮肤。A face image is an image corresponding to the facial area of a person in a person image. The parts contained in the face image can be divided into: facial features and other continuous facial skin except the facial features.
可理解的，人物的眼睛、耳朵、眉毛、嘴巴、以及鼻子均可以称为五官部位。图像中眼睛、耳朵、眉毛、嘴巴、以及鼻子对应的区域均可以称为五官区域。It is understandable that the eyes, ears, eyebrows, mouth, and nose of a person can all be referred to as facial features. The areas corresponding to the eyes, ears, eyebrows, mouth, and nose in an image can all be referred to as facial features areas.
在本申请中，一张人物图像中可以包含有‘眼睛、耳朵、眉毛、鼻子、嘴巴’中的一项或多项五官部位。In the present application, a person image may include one or more facial features of 'eyes, ears, eyebrows, nose, and mouth'.
在本申请中，五官图像为人物图像中包含的每一个五官部位对应区域的图像的集合。In the present application, the facial feature image is a collection of images of the corresponding areas of each facial feature part contained in the person image.
在本申请中，所描述的面部图像为人物图像的人脸图像中除了五官部位之外的其他皮肤区域对应的图像，且该除了五官部位之外的其他皮肤区域中对应的图像是一张连续图像。In the present application, the facial image described is an image corresponding to other skin areas except the facial features in the face image of the person image, and the corresponding image in other skin areas except the facial features is a continuous image.
可理解的，本文以面部图像为连续的一张图像示出，但基于具体的需求和设计，面部图像包含的部位中每两项部位对应的图像可以是连续的也可以是不连续的，本文对此不做限定。It is understandable that this article shows the facial image as a continuous image, but based on specific needs and designs, the images corresponding to every two parts of the parts included in the facial image can be continuous or discontinuous, and this article does not limit this.
在本申请中，电子设备可以基于人像抠图技术获取人物图像中的肖像和/或背景图像，以及基于人脸解析技术获取人物图像中五官图像和/或面部图像。In the present application, the electronic device can obtain a portrait and/or background image in a person image based on a portrait cutout technology, and obtain a facial feature image and/or a face image in a person image based on a face analysis technology.
示例性的，如图1所示，图像A为包含人脸的人物图像，电子设备可以基于人像抠图技术将图像A分割为肖像101和背景图像102，或者也可以理解为将图像A划分为肖像101和背景图像102。以及，可以基于人脸解析技术为肖像101中的人脸区域的五官部位和面部皮肤部位对应的区域位置标上对应的标签，例如为人脸区域中的眉毛103、耳朵104、眼睛105、鼻子106、嘴巴107、面部108分别对应的区域位置标注对应的标签。基于此，电子设备可以获取人物图像的五官图像和面部图像，例如，图像A对应的五官图像包括眉毛103、耳朵104、眼睛105、鼻子106、以及嘴巴107对应的图像的集合，面部图像为面部108对应的图像。Exemplarily, as shown in FIG1 , image A is a person image including a face, and the electronic device can segment image A into a portrait 101 and a background image 102 based on the portrait cutout technology, or it can also be understood as segmenting image A into a portrait 101 and a background image 102. Also, corresponding labels can be marked for the area positions corresponding to the facial features and facial skin parts of the face region in the portrait 101 based on the face analysis technology, for example, corresponding labels are marked for the area positions corresponding to the eyebrows 103, ears 104, eyes 105, nose 106, mouth 107, and face 108 in the face region. Based on this, the electronic device can obtain the facial features image and facial image of the person image, for example, the facial features image corresponding to image A includes a set of images corresponding to eyebrows 103, ears 104, eyes 105, nose 106, and mouth 107, and the facial image is an image corresponding to face 108.
在一种可能的实现方式中，人脸解析还可以为肖像101中的头发、脖子、饰品(例如眼镜、帽子、耳环、项链等)等对应的区域位置标上对应的标签。In a possible implementation, the face analysis may also label the corresponding regions of the hair, neck, accessories (such as glasses, hat, earrings, necklace, etc.) in the portrait 101 with corresponding labels.
上述对概念的介绍可以使用在以下的实施例中。The above introduction to the concepts can be used in the following embodiments.
以下结合其他图像处理方法说明本申请提供的图像处理方法的优势：The advantages of the image processing method provided by this application are explained below in combination with other image processing methods:
在一些其他的图像处理方法中，对于存在人像模糊、人脸细节缺失等的图像质量问题的人物图像，大多使用基于大量非特定人像的图像数据训练得到的图像处理模型对图像进行增强处理。这种方法，由于图像处理模型使用了大量非特定人像的图像数据，缺乏当前特征人像的图像先验信息，未能充分利用当前特定人像的历史先验数据，处理后的人物的特点可能会存在与当前人物身份信息不符的情况，例如，增加了并不属于当前人像的深邃的眼窝、高挺的鼻子山根，或者增强后的人物图像与原始图像的在肤质细节或五官细节上会存在明显的差异，图像增强效果有待提高。In some other image processing methods, for image quality problems such as blurred portraits and missing facial details, most of them use image processing models trained based on a large amount of non-specific portrait image data to enhance the images. This method, because the image processing model uses a large amount of non-specific portrait image data, lacks the image prior information of the current feature portrait, and fails to fully utilize the historical prior data of the current specific portrait, the characteristics of the processed person may be inconsistent with the current person's identity information. For example, deep eye sockets and high nose bridges that do not belong to the current portrait are added, or there are obvious differences in skin texture details or facial features between the enhanced person image and the original image. The image enhancement effect needs to be improved.
有鉴于此，本申请提供一种图像处理方法，提升当前待增强人物图像的面部纹理细节、五官清晰度以及五官细节，以及改善处理后的人物图像中人物的特点包含与当前人物的特点不符的问题。In view of this, the present application provides an image processing method to improve the facial texture details, facial feature clarity and facial feature details of the current person image to be enhanced, and to improve the problem that the characteristics of the person in the processed person image are inconsistent with the characteristics of the current person.
在一种可能的实现方式中，采用本申请提供的图像处理方法对当前待增强人物图像(第一图像)进行增强所采用的方法包括：基于第一图像对应的人物的历史先验信息选取优质图像，基于该优质图像对第一图像的肖像或肖像的局部图像进行清晰度、皮肤纹理细腻度、五官细节的增强处理，得到增强后的肖像或增强后的肖像局部图像，以及对增强后的肖像或增强后的肖像局部图像与第一图像的背景图像进行融合增强，融合增强中包括拼图边缘平滑处理、基于增强后的肖像或肖像局部的色彩信息对背景图像进行色彩校正，增强处理后得到的第二图像中其肖像或肖像局部图像包含特定人像中细腻的纹理，背景图像与肖像或肖像局部图像的色调一致。In one possible implementation, the method used to enhance the current person image to be enhanced (first image) using the image processing method provided by the present application includes: selecting a high-quality image based on historical prior information of the person corresponding to the first image, enhancing the clarity, skin texture fineness, and facial features details of the portrait or a partial image of the portrait of the first image based on the high-quality image to obtain an enhanced portrait or an enhanced partial image of the portrait, and fusing the enhanced portrait or the enhanced partial image of the portrait with the background image of the first image for enhancement, wherein the fusion enhancement includes jigsaw edge smoothing and color correction of the background image based on the color information of the enhanced portrait or the partial portrait, and the portrait or the partial image of the portrait in the second image obtained after the enhancement processing contains the delicate texture of the specific portrait, and the background image has the same color tone as the portrait or the partial image of the portrait.
上述与第一图像匹配的优质图像可以是指该优质图像与第一图像的人脸聚类标签一致、图像质量较高的优质图像，图像质量较高是指该优质图像的面部清晰度好、五官清晰度好、五官细节丰好(例如嘴唇色彩和纹理好、眉毛的毛流感好、鼻子色彩对比度反映的鼻子立体度好)、皮肤纹理细腻度好，该优质图像可以包含优质肖像或优质肖像局部图像。上述优质图像与第一图像的人脸聚类标签一致根据第一图像中包含的肖像图像的数量的不同涵义有所不同，例如，若第一图像仅包含一个肖像图像，则第一图像的人脸聚类标签与该优质图像的人脸聚类标签一致，若第一图像包含两个或两个以上肖像图像，也即第一图像对应两个或两个以上人脸聚类标签，则优质图像对应的人脸聚类标签属于该第一图像对应两个或两个以上人脸聚类标签中的其中一个。The high-quality image matched with the first image may refer to a high-quality image whose face clustering label is consistent with that of the first image and whose image quality is high. The high image quality means that the face definition, facial features definition, details of facial features (e.g., good lip color and texture, good eyebrow hair flow, good nose three-dimensionality reflected by nose color contrast) and skin texture of the high-quality image are good. The high-quality image may include a high-quality portrait or a partial image of a high-quality portrait. The consistency of the face clustering label of the high-quality image and the first image may have different meanings depending on the number of portrait images included in the first image. For example, if the first image only includes one portrait image, the face clustering label of the first image is consistent with that of the high-quality image. If the first image includes two or more portrait images, that is, the first image corresponds to two or more face clustering labels, the face clustering label corresponding to the high-quality image belongs to one of the two or more face clustering labels corresponding to the first image.
由此，采用上述优质图像对第一图像进行增强，利用当前被摄对象的先验知识中画质清晰度好、五官细节好、皮肤纹理细腻度好的优质图像对第一图像进行图像增强处理，将优质图像中清晰且细腻的皮肤纹理特征融合或迁移到第一图像中，得到的第二图像画质清晰度好、皮肤纹理细腻度好，在提升人物图像质量的同时，改善处理后的人物的特点可能会存在与当前人物身份信息不符或增强后的人物图像与原始图像的在肤质细节或五官细节上会存在明显的差异的问题。Therefore, the above-mentioned high-quality image is used to enhance the first image, and the first image is enhanced by utilizing the high-quality image with good picture clarity, good facial details, and good skin texture in the prior knowledge of the current subject. The clear and delicate skin texture features in the high-quality image are fused or migrated to the first image, and the obtained second image has good picture clarity and good skin texture. While improving the quality of the character image, the improved characteristics of the processed character may be inconsistent with the current character identity information, or there may be obvious differences in skin details or facial details between the enhanced character image and the original image.
例如，采用本申请提供的图像处理方法，可以解决实际使用场景中，存在多种因为拍摄情况复杂引发的人像质量问题。例如，在暗光场景中，拍摄得到的第一图像光照不足、图像噪声多、人像模糊、存在局部失真等人像质量问题，采用本申请提供的图像增强方法利用优质图像对第一图像进行增强处理，一方面，优质图像的光照信息充足，纹理细节多，利用优质图像对第一图像的肖像或肖像局部图像进行增强处理，增强后的肖像或肖像局部图像的皮肤纹理会更细腻、色调更自然，可以很好地改善图像噪声多、人像模糊、局部细节涂抹严重等的问题，同时，优质图像的光照充足，增强后的肖像或肖像局部图像中包含的皮肤纹理对应的像素点也是光照充足的，从而还可以改善暗光场景下光照不足、导致图像清晰度欠佳、图像亮度较低的问题。另外，背景融合增强步骤中将增强后的肖像与背景图像进行融合增强时，对拼图边缘做平滑处理，基于增强后的肖像的色彩信息对背景图像进行光照矫正，进一步改善图像的光照质量、以及协调图像整体色差，使得增强得到的第二图像整体更为自然。For example, the image processing method provided by the present application can solve a variety of portrait quality problems caused by complex shooting conditions in actual use scenarios. For example, in a dark light scene, the first image captured has portrait quality problems such as insufficient lighting, high image noise, blurred portraits, and local distortion. The image enhancement method provided by the present application uses a high-quality image to enhance the first image. On the one hand, the high-quality image has sufficient lighting information and many texture details. The portrait or partial portrait image of the first image is enhanced using the high-quality image. The skin texture of the enhanced portrait or partial portrait image will be more delicate and the tone will be more natural, which can well improve the problems of high image noise, blurred portraits, and severe smearing of local details. At the same time, the high-quality image has sufficient lighting, and the pixel points corresponding to the skin texture contained in the enhanced portrait or partial portrait image are also well illuminated, which can also improve the problem of insufficient lighting in dark light scenes, resulting in poor image clarity and low image brightness. In addition, when the enhanced portrait is fused and enhanced with the background image in the background fusion enhancement step, the puzzle edges are smoothed, and the background image is corrected for illumination based on the color information of the enhanced portrait to further improve the image illumination quality and coordinate the overall color difference of the image, so that the enhanced second image is more natural as a whole.
例如，采用本申请提供的图像处理方法，可以改善因曝光控制不准引发人物图像出现欠曝或过曝的图像质量问题，恢复人像真实细节。例如第一图像的左脸颊区域对应的图像存在过曝的图像质量问题，而电子设备检索到的优质图像中也包含左脸颊对应的图像、且其清晰度高、皮肤纹理细节多，则电子设备基于该优质图像对第一图像中的左脸颊进行图像增强，为该存在过曝问题的左脸颊填充相应的细腻的皮肤纹理，改善第一图像存在的过曝问题，提高人物图像的质感和真实感。另外，还可以对增强后的肖像或肖像局部做进一步的曝光矫正，进一步提高图像增强效果。For example, the image processing method provided by the present application can improve the image quality problem of underexposure or overexposure of the character image caused by inaccurate exposure control, and restore the real details of the portrait. For example, the image corresponding to the left cheek area of the first image has an overexposure image quality problem, and the high-quality image retrieved by the electronic device also includes the image corresponding to the left cheek, and the image has high clarity and many skin texture details. Then, the electronic device performs image enhancement on the left cheek in the first image based on the high-quality image, fills the left cheek with the overexposure problem with the corresponding delicate skin texture, improves the overexposure problem in the first image, and improves the texture and realism of the character image. In addition, further exposure correction can be performed on the enhanced portrait or part of the portrait to further improve the image enhancement effect.
同理，采用本申请提供的图像处理方法，也可以改善因数字变焦、对焦不准或拍照抖动等引起的人像模糊、细节缺失或涂抹的图像质量问题，在此不再详述。Similarly, the image processing method provided in this application can also improve image quality problems such as blurred portraits, missing details or smearing caused by digital zoom, inaccurate focus or camera shake, which will not be described in detail here.
在另外一些可能的实现方式中，上述优质图像为优质肖像，且该优质肖像的头部姿态与第一图像的头部姿态差异较大的情况下，电子设备可以基于三维人脸重建或NerfGAN技术对优质肖像进行头部姿态矫正，利用矫正后的肖像图像对第一图像进行增强处理。相比于直接使用与待增强肖像姿态差异较大的优质肖像对第一图像的肖像进行增强处理，可以得到较好的增强效果。In some other possible implementations, when the high-quality image is a high-quality portrait, and the head posture of the high-quality portrait is significantly different from that of the first image, the electronic device can correct the head posture of the high-quality portrait based on 3D face reconstruction or NerfGAN technology, and use the corrected portrait image to enhance the first image. Compared with directly using the high-quality portrait with a posture significantly different from that of the portrait to be enhanced to enhance the portrait of the first image, a better enhancement effect can be obtained.
示例性的，基于三维人脸重建对优质图像进行头部姿态矫正可以包括：电子设备对优质肖像进行3D人脸重建得到三维人脸模型，以第一图像的头部姿态为参照对该三维人脸模型的头部进行姿态矫正，再将矫正后的三维人脸模型渲染为2D人脸图像，基于该2D人脸图像对第一图像的肖像进行图像增强处理。由此，利用优质肖像进行3D人脸重建和头部姿态矫正，可以得到与待增强肖像姿态一致或相近的3D人脸重建结果，并将该3D人脸重建结果生成2D人脸图像，基于该2D图像指导第一图像的肖像进行增强，相比于直接使用与待增强肖像姿态差异较大的上述优质肖像对第一图像的肖像进行增强处理，可以得到较好的增强效果。Exemplarily, performing head posture correction on a high-quality image based on three-dimensional face reconstruction may include: an electronic device performs 3D face reconstruction on a high-quality portrait to obtain a three-dimensional face model, performs posture correction on the head of the three-dimensional face model with reference to the head posture of the first image, and then renders the corrected three-dimensional face model into a 2D face image, and performs image enhancement processing on the portrait of the first image based on the 2D face image. Thus, by performing 3D face reconstruction and head posture correction using the high-quality portrait, a 3D face reconstruction result that is consistent with or similar to the posture of the portrait to be enhanced can be obtained, and a 2D face image is generated from the 3D face reconstruction result, and the portrait of the first image is enhanced based on the 2D image. Compared with directly using the above-mentioned high-quality portrait with a posture that is significantly different from the portrait to be enhanced to enhance the portrait of the first image, a better enhancement effect can be obtained.
在另外一些其他图像处理方法中，使用用户历史‘好’照片修复‘坏’照片，主要流程包含两步，第一步：调整优质历史人物图像的大小为与待增强的原始人物图像相同的大小，该优质历史人物图像与该原始图像包含同一特定人物，且其头部姿态一致，基于传统方法利用大量的优质历史人物图像对当前原始图像的人脸区域进行图像增强。In some other image processing methods, the user's historical 'good' photos are used to repair 'bad' photos. The main process includes two steps. The first step is to adjust the size of the high-quality historical figure image to the same size as the original figure image to be enhanced. The high-quality historical figure image and the original image contain the same specific person and have the same head posture. Based on traditional methods, a large number of high-quality historical figure images are used to enhance the face area of the current original image.
一方面，这种方法未基于增强后的肖像的色彩信息对原始人物图像的背景图像进行光照矫正，增强后的肖像与背景图像的色差整体不协调，增强得到的图像效果不自然。On the one hand, this method does not perform lighting correction on the background image of the original character image based on the color information of the enhanced portrait. The color difference between the enhanced portrait and the background image is not harmonious as a whole, and the enhanced image effect is unnatural.
然而，采用本申请提供的图像处理方法，电子设备可以对增强肖像和该第一图像的背景图像进行背景融合增强，具体包括将背景图像和增强肖像进行图像拼接并对拼接边缘进行平滑处理，以及利用增强肖像的色彩信息对该背景图像进行光照矫正，从而可以减小增强肖像的与背景图像在色彩差异，使得合成后的第二图像更自然。However, by adopting the image processing method provided by the present application, the electronic device can perform background fusion enhancement on the enhanced portrait and the background image of the first image, specifically including stitching the background image and the enhanced portrait and smoothing the stitching edges, and using the color information of the enhanced portrait to perform lighting correction on the background image, thereby reducing the color difference between the enhanced portrait and the background image, making the synthesized second image more natural.
另外一方面，这种方法采用传统方法对脸部区域进行整体增强，未能考虑脸部不同区域的几何和细节的差异，增强效果有待提高。On the other hand, this method uses traditional methods to perform overall enhancement on the facial area, but fails to consider the differences in geometry and details of different facial regions, and the enhancement effect needs to be improved.
而采用本申请实施例提供的图像处理方法，电子设备可以提取人物图像不同区域的特征，上述优质图像可以包括肖像局部图像，例如优质五官图像和优质面部图像，分别利用优质五官图像中的五官特征对第一图像进行五官增强，利用优质面部图像中的面部特征对第一图像进行面部增强，例如对图像中眼睛区域包含的眼神光的轮廓清晰度和亮度做单独处理，对眼睛中的虹膜的纹理和色彩进行增强处理，增强后得到的图像真实感更好，考量脸部不同区域的细节特征的差异，图像增强效果好。By adopting the image processing method provided in the embodiment of the present application, the electronic device can extract features of different areas of the character image. The above-mentioned high-quality image can include a partial portrait image, such as a high-quality facial feature image and a high-quality facial image. The facial features of the high-quality facial feature image are used to enhance the facial features of the first image, and the facial features of the high-quality facial image are used to enhance the face of the first image. For example, the contour clarity and brightness of the eye light contained in the eye area of the image are processed separately, and the texture and color of the iris in the eye are enhanced. The image obtained after enhancement has a better sense of reality, and the image enhancement effect is good considering the differences in the detailed features of different facial areas.
以下结合图2介绍本申请实施例提供的系统图。The following is a system diagram provided in an embodiment of the present application in conjunction with FIG2 .
如图2所示，该系统图包括在线增强子系统10、人物特征检索子系统20、人物处理子系统30、先验知识管理子系统40、以及图像数据库50。As shown in FIG. 2 , the system diagram includes an online enhancement subsystem 10 , a character feature retrieval subsystem 20 , a character processing subsystem 30 , a priori knowledge management subsystem 40 , and an image database 50 .
如图3A所示，增强子系统10包括判断模块和增强融合模块。As shown in FIG3A , the enhancement subsystem 10 includes a judgment module and an enhancement fusion module.
判断模块用于判断人物图像是否满足预设增强条件，也即判断人物图像是否需要进行图像增强。具体的，关于预设增强条件的描述可以参照下文步骤S506中的相关说明，在此不再赘述。The judgment module is used to judge whether the person image meets the preset enhancement condition, that is, to judge whether the person image needs to be enhanced. Specifically, the description of the preset enhancement condition can refer to the relevant description in step S506 below, which will not be repeated here.
增强融合模块用于对人物图像进行增强处理，其主要功能包括：对人物图像中的肖像的整体图像和/或肖像的局部图像进行图像增强，对两项增强后的肖像局部图像之间进行拼图融合，对增强后的图像以及未增强的图像进行融合增强。The enhancement fusion module is used to enhance the character image. Its main functions include: image enhancement of the overall image of the portrait and/or the partial image of the portrait in the character image, puzzle fusion between the two enhanced portrait partial images, and fusion enhancement of the enhanced image and the unenhanced image.
具体的，如图3A中融合增强模块所示，包含五官增强、面部增强、肖像增强、背景融合增强。其中，五官增强是指利用优质五官图像对第一图像的五官图像进行增强处理，关于优质五官图像、以及具体如何基于优质五官图像对第一图像的五官图像进行增强处理可以参照下文图5中步骤S507和S508中的相关描述。面部增强是指利用优质面部图像对第一图像的面部图像进行增强处理，关于优质面部图像、以及具体如何基于优质面部图像对第一图像的面部图像进行增强处理可以参照下文图5中步骤S509和S510中的相关描述，肖像增强是指利用优质肖像对第一图像的肖像图像进行增强处理，关于优质肖像、以及具体如何基于优质肖像对第一图像的肖像图像进行增强处理可以参照下文图6中步骤S606和S608中的相关描述。背景融合增强是指对增强好的肖像图像与背景图像进行融合增强，关于增强好的肖像图像可以参照下文S511中或S608中增强肖像A的相关描述，关于具体如何对增强好的肖像图像与背景图像进行融合增强可以参照下文S512中的相关描述，在此不再详述。Specifically, as shown in the fusion enhancement module in FIG3A, it includes facial enhancement, face enhancement, portrait enhancement, and background fusion enhancement. Among them, facial enhancement refers to enhancing the facial image of the first image using the high-quality facial image. For the high-quality facial image and how to enhance the facial image of the first image based on the high-quality facial image, please refer to the relevant description in steps S507 and S508 in FIG5 below. Facial enhancement refers to enhancing the facial image of the first image using the high-quality facial image. For the high-quality facial image and how to enhance the facial image of the first image based on the high-quality facial image, please refer to the relevant description in steps S509 and S510 in FIG5 below. Portrait enhancement refers to enhancing the portrait image of the first image using the high-quality portrait. For the high-quality portrait and how to enhance the portrait image of the first image based on the high-quality portrait, please refer to the relevant description in steps S606 and S608 in FIG6 below. Background fusion enhancement refers to the fusion enhancement of the enhanced portrait image and the background image. For the enhanced portrait image, please refer to the relevant description of enhancing portrait A in S511 or S608 below. For the specific method of fusion enhancement of the enhanced portrait image and the background image, please refer to the relevant description in S512 below, which will not be described in detail here.
人物特征检索子系统20，用于检索图像数据库50中与相关人物图像对应的数据。示例性的，如图3B所示，其主要功能包括：肖像数据检索、面部数据检索、五官数据检索、人脸属性特征检索。肖像数据为肖像图像(在本文的一些描述中肖像图像也简称为肖像)，面部数据可以是面部图像、面部评分结果；五官数据可以是五官图像、五官评分结果；人脸属性特征参数包含但不限于：头部姿态、人脸本征信息。关于肖像数据检索的描述可以参照下文图6中S607的相关说明，关于面部数据检索的描述可以参照下文图5中S509的相关说明，在此不再详述，关于五官数据检索的描述可以参照下文图5中S507的相关说明，在此不再详述，关于人脸属性特征检索的描述可以参照下文图6中S607中与头部姿态相关的描述以及S611中与人脸本征信息相关的描述。The character feature retrieval subsystem 20 is used to retrieve data corresponding to the relevant character image in the image database 50. Exemplarily, as shown in FIG3B, its main functions include: portrait data retrieval, facial data retrieval, facial features data retrieval, and facial attribute feature retrieval. Portrait data is a portrait image (in some descriptions of this article, portrait image is also referred to as portrait), facial data can be a facial image, a facial scoring result; facial features data can be a facial features image, a facial features scoring result; facial attribute feature parameters include but are not limited to: head posture, facial intrinsic information. For the description of portrait data retrieval, please refer to the relevant description of S607 in FIG6 below, for the description of facial data retrieval, please refer to the relevant description of S509 in FIG5 below, which will not be described in detail here, for the description of facial features data retrieval, please refer to the relevant description of S507 in FIG5 below, which will not be described in detail here, for the description of facial attribute feature retrieval, please refer to the description related to head posture in S607 in FIG6 below and the description related to facial intrinsic information in S611.
示例性的，如图3C所示，人物处理子系统30包含特征提取模块、人像处理模块、评分模块、人脸属性估计模块、3D人脸重建模块。Exemplarily, as shown in FIG3C , the character processing subsystem 30 includes a feature extraction module, a portrait processing module, a scoring module, a face attribute estimation module, and a 3D face reconstruction module.
其中，特征提取模块的主要功能包括：人脸检测、人脸特征提取、人脸纹理特征提取、以及图像特征提取。Among them, the main functions of the feature extraction module include: face detection, face feature extraction, face texture feature extraction, and image feature extraction.
人脸检测可以确定一张图像中是否存在人脸。人脸检测功能在下文图5中的步骤S502中体现，可以用于检测第一图像中是否存在人脸。Face detection can determine whether a face exists in an image. The face detection function is embodied in step S502 in FIG. 5 below, and can be used to detect whether a face exists in a first image.
人脸特征提取包括人脸关键点提取，人脸关键点可以是98点人脸关键点，其中98点人脸关键点包括内部关键点和轮廓关键点，内部关键点包含眉毛、眼睛、鼻子、嘴巴、耳朵区域的关键点。进一步的，人脸关键点还可以包括眼睛部位的瞳孔关键点。人脸特征提取得到的数据可以用于对人物图像中的人脸进行人脸聚类。The facial feature extraction includes the extraction of facial key points. The facial key points can be 98 facial key points, wherein the 98 facial key points include internal key points and contour key points. The internal key points include key points of eyebrows, eyes, nose, mouth, and ear areas. Furthermore, the facial key points can also include pupil key points of the eyes. The data obtained by facial feature extraction can be used to perform facial clustering on the faces in the character images.
关于人脸纹理特征提取中的纹理提取方法可以参照下文步骤S504中关于面部评分标准中介绍的提取面部纹理作为面部图像的评分依据中的相关说明，在此不再详述。Regarding the texture extraction method in facial texture feature extraction, reference may be made to the relevant description of extracting facial texture as a basis for scoring facial images in the facial scoring standard in step S504 below, which will not be described in detail here.
图像特征包括颜色特征、纹理特征、边缘特征以及空间几何特征等。例如，图像特征包括但不限于：尺度不变特征变换(scale-invariant feature transform，SIFT)特征、描述符(Descriptor)特征。图像特征的分析方法可以包括但不限于：主成分分析(principal component analysis，PCA)。关于提取图像特征的相关说明可以参照下文图5所示的步骤S512中提取增强肖像A和背景图像的图像特征的相关描述，在此不再详述。Image features include color features, texture features, edge features, and spatial geometric features. For example, image features include, but are not limited to, scale-invariant feature transform (SIFT) features and descriptor features. Image feature analysis methods may include, but are not limited to, principal component analysis (PCA). For relevant instructions on extracting image features, please refer to the relevant description of extracting image features of the enhanced portrait A and background image in step S512 shown in FIG. 5 below, which will not be described in detail here.
人像处理模块主要功能包括：人脸聚类、人脸解析、人像抠图。The main functions of the portrait processing module include: face clustering, face analysis, and portrait cutout.
人脸聚类可以将属于同一个人的图像归为一类，为肖像标上对应的人物聚类标签。Face clustering can group images belonging to the same person into one category and label the portraits with corresponding person cluster labels.
人脸解析可以给人物图像的肖像中的人脸区域的五官部位和面部皮肤部位对应的区域位置标上对应的标签，各个部位和部件对应的区域位置标注对应的标签，包括眼睛、鼻子、嘴巴、眉毛、耳朵、面部、头发、脖子、饰品等区域，例如可以参照图1所示的103至107所示的人脸解析后五官部位的区域划分示意图。Face analysis can label the corresponding regional positions of the facial features and facial skin parts of the face area in the portrait of the person image with corresponding labels. The regional positions corresponding to each part and component are marked with corresponding labels, including the eyes, nose, mouth, eyebrows, ears, face, hair, neck, accessories and other areas. For example, refer to the regional division schematic diagram of the facial features after face analysis shown in 103 to 107 in Figure 1.
人脸抠图可以将人物图像中的肖像和背景图像分割开，例如可以参照图1所示的101和102所示的人像抠图后肖像和背景图像的语义分割mask示意图。Face cutout can separate the portrait and background image in a person image. For example, reference can be made to the semantic segmentation mask schematic diagrams of the portrait and background image after face cutout shown in 101 and 102 in FIG. 1 .
关于人脸聚类、人脸解析、人像抠图相应得到的数据可以参照下文图5中阶段2相关步骤的描述，在此不再详述。The data obtained from face clustering, face analysis, and portrait cutout can be found in the description of the relevant steps of stage 2 in FIG. 5 below, and will not be described in detail here.
评分模块的主要功能包括：The main functions of the scoring module include:
1)画质评分：对人物图像的画质进行评分，得到人物图像的画质评分结果。1) Image quality scoring: Score the image quality of the person image to obtain the image quality scoring result of the person image.
关于画质评分的评分标准以及评分区域可以参照下文图6中步骤S604中的相关说明，在此不再详述。Regarding the scoring criteria and scoring area for image quality scoring, please refer to the relevant description in step S604 in Figure 6 below, which will not be described in detail here.
2)面部评分：对人物图像的面部图像进行评分，得到人物图像的面部评分结果。2) Face scoring: Score the face image of the person image to obtain the face scoring result of the person image.
3)五官评分：对人物图像的五官图像进行评分，得到人物图像的五官评分结果。3) Facial features scoring: Score the facial features of the person image to obtain the facial features scoring result of the person image.
在本申请中，对一张人物图像进行五官评分，是指分别对该人物图像中包含的每一项五官图像进行评分，并得到包含多项评分数值的五官评分结果。例如，一张人物图像的五官评分结果包括：眉毛评分结果、眼睛评分结果、鼻子评分结果、嘴唇评分结果、耳朵评分结果；其中，眉毛评分结果、鼻子评分结果、嘴唇评分结果、以及耳朵评分结果对应的评分标准为图像清晰度，眼睛评分结果对应的评分标准为图像清晰度和眼睛睁开程度。In this application, scoring the facial features of a person image means scoring each facial feature image contained in the person image separately, and obtaining a facial feature scoring result containing multiple scoring values. For example, the facial feature scoring result of a person image includes: eyebrow scoring result, eye scoring result, nose scoring result, lip scoring result, and ear scoring result; wherein the scoring criteria corresponding to the eyebrow scoring result, nose scoring result, lip scoring result, and ear scoring result are image clarity, and the scoring criteria corresponding to the eye scoring result are image clarity and eye openness.
关于面部评分和五官评分的评分标准可以参照下文图5中步骤S504中的相关说明，在此不再详述。Regarding the scoring criteria for facial scoring and facial features scoring, please refer to the relevant instructions in step S504 in Figure 5 below, which will not be described in detail here.
人脸属性估计模块的主要功能包括：头部姿态估计、人脸本征估计。The main functions of the face attribute estimation module include: head posture estimation and face intrinsic estimation.
1)头部姿态估计：对人物图像中的肖像进行姿态估计得到人像头部的欧拉旋转角；例如获取头部姿态欧拉角的方法包括但不限于头部姿态估计(head pose estimation)算法。1) Head pose estimation: The pose of the portrait in the person image is estimated to obtain the Euler rotation angle of the head of the portrait; for example, methods for obtaining the Euler angle of the head pose include but are not limited to the head pose estimation algorithm.
2)人脸本征估计：对人物图像进行人脸本征估计得到人脸本征信息。人脸本征信息是指人脸本质上固有的特征，可以基于逆渲染计算得到。在本申请中，人脸本征信息包括但不限于：漫反射、镜面反射、法线(体现人脸几何信息)、高光参数。2) Face intrinsic estimation: Face intrinsic estimation is performed on the character image to obtain face intrinsic information. Face intrinsic information refers to the inherent features of the face, which can be obtained based on inverse rendering calculation. In this application, face intrinsic information includes but is not limited to: diffuse reflection, specular reflection, normal (reflecting face geometric information), and highlight parameters.
3D人脸重建模块的功能是对人物图像中的人脸进行3D人脸建模得到人脸三维模型，再将人脸三维模型渲染为2D图像，输出具备三维信息的2D图像。将人脸三维模型转为2D图像的方法包括但不限于神经渲染(neural rendering)技术，基于神经渲染将人脸三维模型渲染成2D图像，或者也可以是其他可以达到相同效果的算法或模型，在此不再详述。The function of the 3D face reconstruction module is to perform 3D face modeling on the face in the character image to obtain a 3D face model, then render the 3D face model into a 2D image, and output a 2D image with 3D information. The method of converting the 3D face model into a 2D image includes but is not limited to neural rendering technology, which renders the 3D face model into a 2D image based on neural rendering, or other algorithms or models that can achieve the same effect, which will not be described in detail here.
关于3D人脸重建模块的具体实现可以参照图6中步骤S611的相关说明。For the specific implementation of the 3D face reconstruction module, please refer to the relevant description of step S611 in Figure 6.
图像数据库50用于存储人物图像的相关数据，人物图像的相关数据包括但不限于：索引(该索引是指对人物图像的唯一标识)、原始图像、人像抠图数据、人脸解析数据、面部评分结果、五官评分结果、人脸聚类标签、头部姿态、人脸本征信息中的一项或多项。其中人像抠图可以是人像抠图mask，包括人物图像的肖像和背景图像；人脸解析数据可以是人脸解析mask，包括但不限于人物图像中肖像的五官图像和面部图像，在一种可能的实现方式中，人脸解析mask还可以包括人物图像中肖像的脖子、头发、饰品等对应的图像。The image database 50 is used to store data related to person images, including but not limited to: index (the index refers to a unique identifier of the person image), original image, portrait cutout data, face analysis data, facial scoring results, facial features scoring results, face clustering labels, head posture, and one or more of facial intrinsic information. The portrait cutout can be a portrait cutout mask, including a portrait and background image of the person image; the face analysis data can be a face analysis mask, including but not limited to facial features and facial images of the portrait in the person image. In a possible implementation, the face analysis mask can also include images corresponding to the neck, hair, accessories, etc. of the portrait in the person image.
在一种可能的实现方式中，人物图像的相关数据还可以包括上述拍摄模式标签和画质评分结果。In a possible implementation, the relevant data of the person image may also include the above-mentioned shooting mode label and image quality rating result.
在一种可能的实现方式中，人物图像(图像X.jpg)中包含的人脸的数目大于或等于2，图像数据库50中存储的该人物图像的相关数据如下表1所示。In a possible implementation, the number of faces included in the person image (image X.jpg) is greater than or equal to 2, and the relevant data of the person image stored in the image database 50 is shown in Table 1 below.
表1Table 1
先验知识管理子系统40用于建立和维护图像数据库50。其中，先验知识管理子系统40可以为人物图像创建索引，并将人物图像相关数据和该人物图像的索引的关联关系存储到图像数据库50中，基于该索引从图像数据库50中检索对应的人物图像的数据。The prior knowledge management subsystem 40 is used to establish and maintain an image database 50. The prior knowledge management subsystem 40 can create an index for a person image, store the association between the person image-related data and the index of the person image in the image database 50, and retrieve the corresponding person image data from the image database 50 based on the index.
可理解的，电子设备作为客户端可以向云服务器端请求数据，上述增强子系统10、人物特征检索子系统20、人物处理子系统30、先验知识管理子系统40以及图像数据库50，均可以是电子设备端中与本申请提供的图像处理方法相关的功能模块。在另外一些实现方式中，增强子系统10、人物特征检索子系统20、人物处理子系统30、先验知识管理子系统40以及图像数据库50中的一项或多项，也可以是电子设备的云服务器端中与本申请提供的图像处理方法相关的功能模块，本文对此不做限定。It is understandable that the electronic device as a client can request data from the cloud server, and the above-mentioned enhancement subsystem 10, character feature retrieval subsystem 20, character processing subsystem 30, prior knowledge management subsystem 40 and image database 50 can all be functional modules related to the image processing method provided by the present application in the electronic device. In some other implementations, one or more of the enhancement subsystem 10, character feature retrieval subsystem 20, character processing subsystem 30, prior knowledge management subsystem 40 and image database 50 can also be functional modules related to the image processing method provided by the present application in the cloud server of the electronic device, which is not limited in this article.
以下结合上述系统图中的各个子系统简要概述本申请实施例提供的图像处理方法。The following briefly summarizes the image processing method provided in the embodiment of the present application in combination with the various subsystems in the above system diagram.
示例性的，电子设备获取到第一图像后，可以调用先验知识管理子系统40为该第一图像建立索引，将该第一图像存储到图像数据库50中，以及，调用增强子系统10中的判断模块，确定第一图像是否满足人像模糊、人像模糊、人脸纹理细节缺失、人脸区域图像噪声高、人脸区域光照不足、人脸区域光照过亮中的一项或多项增强条件；在确定第一图像满足增强条件的情况下，电子设备调用人物特征检索子系统20检索与第一图像匹配的优质图像(例如优质图像可以是优质肖像、优质五官或优质面部)，再基于增强子系统10中的融合增强模块利用该优质图像对第一图像进行图像增强，得到增强后的第一图像。Exemplarily, after the electronic device acquires the first image, it can call the prior knowledge management subsystem 40 to create an index for the first image, store the first image in the image database 50, and call the judgment module in the enhancement subsystem 10 to determine whether the first image satisfies one or more enhancement conditions of blurred portrait, blurred portrait, missing facial texture details, high image noise in the face area, insufficient lighting in the face area, and excessive lighting in the face area; when it is determined that the first image satisfies the enhancement conditions, the electronic device calls the character feature retrieval subsystem 20 to retrieve a high-quality image that matches the first image (for example, the high-quality image can be a high-quality portrait, high-quality facial features, or high-quality face), and then uses the high-quality image to enhance the first image based on the fusion enhancement module in the enhancement subsystem 10 to obtain an enhanced first image.
以下结合图4简要概述本申请实施例提供的图像处理方法的方法流程示意图。如图4所示，所述方法包括：The following is a schematic diagram of a method flow of an image processing method provided by an embodiment of the present application, briefly described in conjunction with FIG4 . As shown in FIG4 , the method includes:
S401，电子设备获取第一图像。S401, the electronic device acquires a first image.
在本申请实施例中，第一图像包含肖像图像。示例性的，电子设备可以基于上述人物处理子系统30中的人脸检测功能确定第一图像中是否包含肖像图像。In the embodiment of the present application, the first image includes a portrait image. Exemplarily, the electronic device can determine whether the first image includes a portrait image based on the face detection function in the above-mentioned character processing subsystem 30.
在本申请实施例中，在第一图像中包含人脸图像、且第一图像满足预设增强条件中的至少一项的情况下，执行步骤S402；其中，该预设增强条件包括以下一项或一项以上：人像模糊、人脸纹理细节缺失、人脸区域图像噪声高、人脸区域光照不足、人脸区域光照过亮。In an embodiment of the present application, when the first image includes a face image and the first image satisfies at least one of the preset enhancement conditions, step S402 is executed; wherein the preset enhancement condition includes one or more of the following: blurred portrait, missing face texture details, high image noise in the face area, insufficient lighting in the face area, and excessive lighting in the face area.
S402，电子设备获取与第一图像匹配的优质图像。S402: The electronic device obtains a high-quality image matching the first image.
在本申请实施例中，优质图像来源于目标人物且第一图像中包含该目标人物的肖像图像。示例性的，第一图像中包含人脸图像A，优质图像中包含人脸图像B，且该人脸图像A的人脸聚类标签与该人脸图像B的人脸聚类标签一致。In an embodiment of the present application, the high-quality image is derived from a target person and the first image includes a portrait image of the target person. Exemplarily, the first image includes a face image A, the high-quality image includes a face image B, and the face clustering label of the face image A is consistent with the face clustering label of the face image B.
在本申请实施例中，优质图像的图像质量优于第一图像的图像质量。In the embodiment of the present application, the image quality of the high-quality image is better than the image quality of the first image.
示例性的，优质图像的图像质量优于第一图像的图像质量包括：优质图像的全局或局部清晰度、人脸皮肤纹理、以及五官细节优于第一图像，该五官细节包括以下一项或一项以上：五官立体感、嘴唇纹理、眼神光的光斑清晰度、眼睛睁开程度、嘴巴张开程度。Exemplarily, the image quality of the high-quality image being better than the image quality of the first image includes: the global or local clarity, facial skin texture, and facial features details of the high-quality image are better than those of the first image, and the facial features details include one or more of the following: three-dimensional sense of facial features, lip texture, clarity of the spot of eye light, degree of eye openness, and degree of mouth openness.
示例性的，上述优质图像包括优质面部图像和第一优质五官图像，该优质面部图像的面部清晰度(或者也可以理解为面部纹理细节)优于第一图像的面部图像，该第一优质五官图像的五官清晰度、五官立体感以及纹理细节(例如嘴唇纹理)优于第一图像的五官图像。关于第一优质五官图像、优质面部图像的说明具体可以参照下文图5中步骤S507和步骤S509中的相关描述，在此不再详述。Exemplarily, the above-mentioned high-quality image includes a high-quality facial image and a first high-quality facial features image, the facial clarity (or facial texture details) of the high-quality facial image is better than that of the facial image of the first image, and the facial features clarity, facial features stereoscopic effect and texture details (such as lip texture) of the first high-quality facial features image are better than those of the facial features image of the first image. For the description of the first high-quality facial features image and the high-quality facial image, please refer to the relevant descriptions in step S507 and step S509 in FIG. 5 below, which will not be described in detail here.
示例性的，上述优质图像包括优质肖像，该优质肖像的画质清晰度、面部清晰度、五官清晰度、人脸皮肤纹理、五官立体感、以及嘴唇纹理优于所述第一图像的肖像图像。在一些可能的实现方式中，该优质图像还可以包括第二优质五官图像，该第二优质五官图像可以包括鼻子图像、嘴巴图像、眉毛图像。关于优质肖像的说明具体可以参照下文图6中步骤S607中的相关描述，在此不再详述，关于第二优质五官图像的说明可以参照下文图6中步骤S616中的相关说明，在此不再详述。Exemplarily, the above-mentioned high-quality image includes a high-quality portrait, and the image clarity, facial clarity, facial feature clarity, facial skin texture, three-dimensional sense of facial features, and lip texture of the high-quality portrait are better than the portrait image of the first image. In some possible implementations, the high-quality image may also include a second high-quality facial feature image, and the second high-quality facial feature image may include a nose image, a mouth image, and an eyebrow image. For the description of the high-quality portrait, please refer to the relevant description in step S607 in Figure 6 below, which will not be described in detail here. For the description of the second high-quality facial feature image, please refer to the relevant description in step S616 in Figure 6 below, which will not be described in detail here.
在一种可能的实现方式中，若电子设备未能获取到包含上述目标人物的肖像图像的历史先验图像，或者电子设备未能获取到包含上述目标人物的肖像图像对应的优质的图像(也即无法获取优质图像)，则电子设备将上述第一图像作为历史图像存储，并在电子设备可以获取到符合条件的优质图像之后，电子设备确定当前电量是否大于或等于预设电量阈值且设备处于充电灭屏状态的情况下，电子设备对该图像A进行图像增强。或者，在电子设备可以获取到符合条件的优质图像之后，电子设备接收到用户主动触发请求对第一图像进行图像增强的指令后，电子设备对该第一图像进行图像增强。In a possible implementation, if the electronic device fails to obtain a historical prior image containing the portrait image of the target person, or the electronic device fails to obtain a high-quality image corresponding to the portrait image of the target person (i.e., it cannot obtain a high-quality image), the electronic device stores the first image as a historical image, and after the electronic device can obtain a qualified high-quality image, the electronic device determines whether the current power is greater than or equal to the preset power threshold and the device is in a charging and screen-off state, and the electronic device performs image enhancement on the image A. Alternatively, after the electronic device can obtain a qualified high-quality image, the electronic device performs image enhancement on the first image after receiving an instruction from the user to actively trigger a request to enhance the first image.
在一种可能的实现方式中，若电子设备接收到用户主动触发的关于第一图像的图像增强请求后，电子设备未能获取到与第一图像匹配的优质图像时，电子设备还可以输出提示信息，该提示信息用于表示未存储有与当前图像匹配的历史先验数据，或者，用于表示未获取到比该第一图像A的图像质量更好的图像，暂时无法进行图像增强处理。In one possible implementation, if the electronic device fails to obtain a high-quality image that matches the first image after receiving an image enhancement request actively triggered by the user, the electronic device may further output a prompt message, wherein the prompt message is used to indicate that no historical prior data matching the current image is stored, or is used to indicate that an image with better image quality than the first image A has not been obtained and image enhancement processing is temporarily unavailable.
S403，电子设备利用优质图像对第一图像中的肖像图像的整体图像和/或局部图像进行图像增强处理，得到第二图像。S403: The electronic device uses the high-quality image to perform image enhancement processing on the entire image and/or a partial image of the portrait image in the first image to obtain a second image.
示例性的，电子设备利用优质图像对第一图像中的肖像图像的整体图像和/或局部图像进行图像增强处理，得到增强肖像，再对该增强肖像和第一图像的原始背景图像进行融合处理，得到与第一图像对应的上述第二图像。在一些可能的实现方式中，增强肖像与原始背景图像之间满足色彩不均匀条件或满足光照不协调条件的情况下，电子设备还可以利用该增强肖像对第一图像的原始背景图像进行色彩校正，具体可以是在该增强肖像和第一图像的原始背景图像进行融合处理之后进行色彩校正，得到上述第二图像。或者，也可以是对利用该增强肖像对第一图像的原始背景图像进行色彩校正之后，再利用该增强肖像和色彩校正后的原始背景图像进行融合处理，得到上述第二图像，从而可以协调图像的整体色差，使得增强得到的第二图像整体更为自然。Exemplarily, the electronic device uses the high-quality image to perform image enhancement processing on the overall image and/or partial image of the portrait image in the first image to obtain an enhanced portrait, and then fuses the enhanced portrait with the original background image of the first image to obtain the above-mentioned second image corresponding to the first image. In some possible implementations, when the enhanced portrait and the original background image meet the color unevenness condition or the lighting incoordination condition, the electronic device can also use the enhanced portrait to perform color correction on the original background image of the first image, specifically, the enhanced portrait and the original background image of the first image are fused and then color corrected to obtain the above-mentioned second image. Alternatively, it is also possible to use the enhanced portrait to perform color correction on the original background image of the first image, and then fuse the enhanced portrait and the color-corrected original background image to obtain the above-mentioned second image, so that the overall color difference of the image can be coordinated, so that the enhanced second image is more natural as a whole.
关于具体如何对第一图像的原始背景图像进行色彩校正以及该增强肖像和第一图像的原始背景图像进行融合处理可以参照下文步骤S512中的相关说明，在此不再详述。For details on how to perform color correction on the original background image of the first image and how to fuse the enhanced portrait with the original background image of the first image, please refer to the relevant instructions in step S512 below, which will not be described in detail here.
在一种可能的实现方式中，优质图像包含第一优质五官图像和优质面部图像，电子设备具体如何获取该优质面部图像和优质五官图像以及如何基于该优质五官图像和优质面部图像对第一图像进行增强获得上述第一图像可以参照下文图5相关说明，在此不再详述。In one possible implementation, the high-quality image includes a first high-quality facial feature image and a high-quality facial image. How the electronic device specifically obtains the high-quality facial image and the high-quality facial feature image and how it enhances the first image based on the high-quality facial feature image and the high-quality facial image to obtain the above-mentioned first image can be referred to the relevant description of Figure 5 below and will not be described in detail here.
在一种可能的实现方式中，优质图像包含优质肖像，或者该优质图像包含优质肖像和第二优质五官图像，电子设备具体如何获取该优质肖像、第二优质五官图像以及如何基于该优质肖像、第二优质五官图像对第一图像进行增强获得上述第一图像可以参照下文图6相关说明，在此不再详述。In one possible implementation, the high-quality image includes a high-quality portrait, or the high-quality image includes a high-quality portrait and a second high-quality facial features image. How the electronic device specifically obtains the high-quality portrait and the second high-quality facial features image and how to enhance the first image based on the high-quality portrait and the second high-quality facial features image to obtain the above-mentioned first image can be referred to the relevant description of Figure 6 below and will not be described in detail here.
以下结合图5所示的方法流程图详细说明上述图2所示的图像处理方法中利用优质面部图像和第一优质五官图像对第一图像进行增强处理的一种具体实现方式。如图5所示，该图像处理方法可以包括以下几个阶段和对应的方法步骤。A specific implementation of enhancing the first image using the high-quality facial image and the first high-quality facial features image in the image processing method shown in Figure 2 is described in detail below in conjunction with the method flow chart shown in Figure 5. As shown in Figure 5, the image processing method may include the following stages and corresponding method steps.
阶段1：电子设备获取到第一图像后，确定第一图像是否包含人脸，在确定包含人脸的情况下进入阶段2。Stage 1: After the electronic device acquires the first image, it determines whether the first image contains a face, and enters stage 2 if it is determined that the first image contains a face.
S501，电子设备获取第一图像。S501: The electronic device acquires a first image.
例如，电子设备获取第一图像的方式包括但不限于：基于相机应用拍摄获取、基于数据传输方式获取，例如数据传输方式可以是网页下载、云盘下载、在即时通讯方式中交互的图像数据中下载、或者，短距离数据传输(例如蓝牙、NFC等)方式，本文对此不做限定。For example, the electronic device obtains the first image in a manner including but not limited to: obtaining by shooting with a camera application, obtaining by data transmission, for example, the data transmission method may be downloading from a web page, downloading from a cloud disk, downloading from image data exchanged in an instant messaging method, or short-distance data transmission (such as Bluetooth, NFC, etc.), which is not limited in this document.
S502，电子设备基于人脸检测确定第一图像是否存在人脸。S502: The electronic device determines whether there is a face in the first image based on face detection.
人脸检测方法包括但不限于旷视Face++人脸检测算法、深度学习方法。具体所使用的人脸检测算法和具体的检测方法为本领域技术人员熟知的技术，在此不再详述。The face detection method includes but is not limited to the Face++ face detection algorithm and the deep learning method. The specific face detection algorithm and the specific detection method used are well known to those skilled in the art and will not be described in detail here.
在确定第一图像存在人脸的情况下，电子设备执行步骤S503、S504、S505；在确定第一图像中不存在人脸的情况下，电子设备不解析第一图像也不对第一图像进行增强处理(在图5中以步骤S514示出)，也即不执行步骤S503至S512。When it is determined that there is a face in the first image, the electronic device executes steps S503, S504, and S505; when it is determined that there is no face in the first image, the electronic device does not parse the first image or perform enhancement processing on the first image (shown as step S514 in Figure 5), that is, does not execute steps S503 to S512.
阶段2：电子设备获取第一图像的相关数据，具体包括：人脸聚类标签、人像抠图数据(包含肖像和背景图像)、人脸解析数据(包含待增强面部图像、待增强五官图像)、面部评分结果、以及五官评分结果。Stage 2: The electronic device obtains relevant data of the first image, specifically including: face clustering labels, portrait cutout data (including portrait and background images), face analysis data (including facial images to be enhanced and facial features images to be enhanced), face scoring results, and facial features scoring results.
S503，电子设备获取第一图像对应的待增强五官图像、待增强面部图像、背景图像。S503, the electronic device obtains the facial features image to be enhanced, the face image to be enhanced, and the background image corresponding to the first image.
例如，电子设备通过上述人物处理子系统30中的人像处理模块的人脸解析和人像抠图功能获取第一图像对应的待增强五官图像、待增强面部图像、肖像、背景图像。其中，人像抠图可以获取第一图像中包含的肖像和除了肖像之外的背景图像，人脸解析可以获取第一图像的待增强五官图像和待增强面部图像。For example, the electronic device obtains the facial features image to be enhanced, the facial image to be enhanced, the portrait, and the background image corresponding to the first image through the facial analysis and portrait cutout functions of the portrait processing module in the above-mentioned character processing subsystem 30. Among them, the portrait cutout can obtain the portrait contained in the first image and the background image other than the portrait, and the facial analysis can obtain the facial features image to be enhanced and the facial image to be enhanced of the first image.
在本申请中，待增强五官图像为第一图像中包含的每一个五官部位对应区域的图像的集合，待增强面部图像为第一图像中的人脸区域中除了五官部位对应的图像之外的其他皮肤区域对应的图像。也可以理解为，面部图像为人物图像中除了五官部位对应区域的图像之外的其他图像，具体还可以参照上述术语解析中与‘五官图像’和‘面部图像’的相关说明，在此不再详述。示例性的，第一图像为如图1所示的图像，则待增强五官图像包括：眉毛103对应的图像、耳朵104对应的图像、眼睛105对应的图像、鼻子106对应的图像、嘴巴107对应的图像，待增强面部图像包括：面部108对应的图像。In the present application, the facial features images to be enhanced are a set of images of the corresponding areas of each facial feature contained in the first image, and the facial images to be enhanced are images corresponding to other skin areas in the face area of the first image except for the images corresponding to the facial features. It can also be understood that the facial images are other images in the character image except for the images of the corresponding areas of the facial features. For details, please refer to the relevant explanations of "facial features images" and "facial images" in the above-mentioned term analysis, which will not be described in detail here. Exemplarily, the first image is the image shown in Figure 1, then the facial features images to be enhanced include: images corresponding to eyebrows 103, images corresponding to ears 104, images corresponding to eyes 105, images corresponding to nose 106, images corresponding to mouth 107, and the facial images to be enhanced include: images corresponding to face 108.
在本申请中，电子设备获取到的待增强五官图像、待增强面部图像、肖像、以及背景图像后，可以将其存储在本地存储空间中，也可以存储到云端(电子设备的云服务器端)，本文对此不做限定。示例性的，人像处理模块获取到第一图像的待增强五官图像、待增强面部图像、肖像、背景图像之后，将其存储到图像数据库50中。In the present application, after the electronic device obtains the facial features image to be enhanced, the facial image to be enhanced, the portrait, and the background image, it can store them in a local storage space or in the cloud (the cloud server of the electronic device), which is not limited in this article. Exemplarily, after the portrait processing module obtains the facial features image to be enhanced, the facial image to be enhanced, the portrait, and the background image of the first image, it stores them in the image database 50.
S504，电子设备获取第一图像的面部评分结果和五官评分结果。S504, the electronic device obtains the face scoring result and the facial features scoring result of the first image.
在本申请中，第一图像的面部评分结果即为上述待增强面部图像的面部评分结果，第一图像的五官评分即为上述待增强五官图像的五官评分结果。示例性的，电子设备通过人物处理子系统30中的评分模块对上述待增强面部图像、待增强五官图像分别进行面部评分和五官评分，以获取该待增强面部图像的面部评分结果和该待增强五官图像的五官评分结果。在本申请实施例中，电子设备在接收到评分模块发送的该面部评分结果和五官评分结果后，可以将该第一图像的面部评分结果和五官评分结果存储到电子设备的本地存储空间中或电子设备的云服务器端中，例如可以是上述图像数据库50。或者，也可以是评分模块在计算得到该第一图像的面部评分结果和五官评分结果后，将该第一图像的面部评分结果和五官评分结果存储到电子设备的本地存储空间中或电子设备的云服务器端，本文对此不做限定。In the present application, the facial scoring result of the first image is the facial scoring result of the facial image to be enhanced, and the facial features scoring of the first image is the facial features scoring result of the facial features image to be enhanced. Exemplarily, the electronic device performs facial scoring and facial features scoring on the facial image to be enhanced and the facial features image to be enhanced respectively through the scoring module in the character processing subsystem 30 to obtain the facial scoring result of the facial image to be enhanced and the facial features scoring result of the facial features image to be enhanced. In an embodiment of the present application, after receiving the facial scoring result and the facial features scoring result sent by the scoring module, the electronic device can store the facial scoring result and the facial features scoring result of the first image in the local storage space of the electronic device or in the cloud server of the electronic device, for example, the image database 50. Alternatively, the scoring module can also store the facial scoring result and the facial features scoring result of the first image in the local storage space of the electronic device or in the cloud server of the electronic device after calculating the facial scoring result and the facial features scoring result of the first image, which is not limited herein.
在本申请实施例中，面部评分的评分标准与人物图像中面部图像的纹理相关，面部图像的纹理细节可以反映人脸皮肤的清晰度、细腻度以及质感。In the embodiment of the present application, the scoring criteria for facial scoring are related to the texture of the facial image in the character image. The texture details of the facial image can reflect the clarity, fineness and texture of the human face skin.
在本申请实施例中，面部评分结果可以是面部纹理特征的纹理值，也可以是将纹理值转换为百分制或等级的结果，本文对此不做限定。In the embodiment of the present application, the facial scoring result may be a texture value of a facial texture feature, or may be a result of converting the texture value into a percentage or grade, which is not limited herein.
以下介绍如何提取第一图像的面部图像的纹理特征作为第一图像的面部评分结果或作为第一图像的面部评分的依据。The following describes how to extract the texture features of the facial image of the first image as the facial scoring result of the first image or as the basis for the facial scoring of the first image.
示例性的，可以通过对图像的人脸纹理进行分析，设计相应的反映人脸纹理状况的特征参数，通过人脸图像中像素值提取相关特征参数反映该人脸图像的纹理。Exemplarily, the facial texture of the image can be analyzed to design corresponding feature parameters reflecting the facial texture condition, and the texture of the facial image can be reflected by extracting relevant feature parameters through pixel values in the facial image.
例如，电子设备可以基于待增强面部图像的LBP模式、灰度共生矩阵、分形维数、Tumera中的一项或多项计算待增强面部图像的纹理特征，以进行面部评分。具体的，电子设备可以基于面部图像的灰度共生矩阵、局部二值模式(local binary pattern，LBP)、分形维数、Tumera中的一项或多项数据计算面部图像的纹理特征。例如可以计算面部图像的灰度共生矩阵在不同维度上的纹理特征，包括均值、方差、熵、以及相关性等，其中方差可以反映纹理波动情况、均值表示纹理的一般程度、熵表示图像中纹理的复杂程度、相关性表示相应的区域的纹理是否相关。再基于该灰度共生矩阵计算得到的一个或多个参数(例如均值、方差、以及相关性)、面部图像的LBP特征、分形维数特征、Tumera特征中的两项或两项以上特征进行PCA降维计算，例如降维至3个特征，再分别对三个特征计算特征贡献率，选择贡献率最大的特征代表人脸图像的纹理信息作为面部图像的纹理特征的评分结果或作为面部图像优劣的评分依据。For example, the electronic device can calculate the texture features of the facial image to be enhanced based on one or more of the LBP mode, gray level co-occurrence matrix, fractal dimension, and Tumera of the facial image to be enhanced to perform facial scoring. Specifically, the electronic device can calculate the texture features of the facial image based on the gray level co-occurrence matrix, local binary pattern (LBP), fractal dimension, and Tumera of the facial image. For example, the texture features of the gray level co-occurrence matrix of the facial image in different dimensions can be calculated, including mean, variance, entropy, and correlation, etc., where the variance can reflect the texture fluctuation, the mean represents the general degree of texture, the entropy represents the complexity of the texture in the image, and the correlation represents whether the textures of the corresponding areas are related. Then, based on one or more parameters (such as mean, variance, and correlation) calculated from the gray level co-occurrence matrix, LBP features of the facial image, fractal dimension features, and two or more of the Tumera features, PCA dimensionality reduction calculation is performed, for example, the dimensionality is reduced to 3 features, and then the feature contribution rates of the three features are calculated respectively, and the feature with the largest contribution rate is selected to represent the texture information of the facial image as the scoring result of the texture feature of the facial image or as the basis for scoring the quality of the facial image.
关于基于人脸图像的灰度共生矩阵、LBP、分形维数、Tumera提取人脸图像在不同维度上的纹理特征仅为示例，还可以基于其他领域技术人员熟知的纹理特征提取技术对人脸图像进行面部或其他对应的区域位置进行纹理特征提取，本文对此不做限定。The extraction of texture features of facial images in different dimensions based on gray-level co-occurrence matrix, LBP, fractal dimension, and Tumera are only examples. Texture features of facial images or other corresponding area positions can also be extracted based on texture feature extraction techniques well known to technicians in other fields. This article does not limit this.
可理解的，纹理的细腻度和清晰度是对应的，在本申请实施例中，面部图像的纹理值可以用于体现面部图像的图像清晰度和图像的纹理细节丰富度。It is understandable that the fineness and clarity of texture correspond to each other. In the embodiment of the present application, the texture value of the facial image can be used to reflect the image clarity of the facial image and the richness of the texture details of the image.
关于基于人脸图像的灰度共生矩阵、LBP、分形维数、Tumera提取人脸图像在不同维度上的纹理特征仅为示例，还可以基于其他领域技术人员熟知的纹理特征提取技术对人脸图像进行面部或其他对应的区域位置进行纹理特征提取，本文对此不做限定。The extraction of texture features of facial images in different dimensions based on gray-level co-occurrence matrix, LBP, fractal dimension, and Tumera are only examples. Texture features of facial images or other corresponding area positions can also be extracted based on texture feature extraction techniques well known to technicians in other fields. This article does not limit this.
可理解的，电子设备可以基于人物处理子系统30中的人脸纹理特征提取模块提取面部图像的面部纹理特征作为面部评分结果或面部评分依据。It is understandable that the electronic device can extract the facial texture features of the facial image based on the facial texture feature extraction module in the character processing subsystem 30 as the facial scoring result or facial scoring basis.
在本申请中，人物图像对应的五官评分结果包括多项评分数值，具体可以包括：眉毛图像的眉毛评分结果、眼睛图像的眼睛评分结果、鼻子图像的鼻子评分结果、嘴巴图像的嘴巴评分结果、以及耳朵图像的耳朵评分结果。In the present application, the facial features scoring results corresponding to the character image include multiple scoring values, which may specifically include: the eyebrow scoring results of the eyebrow image, the eye scoring results of the eye image, the nose scoring results of the nose image, the mouth scoring results of the mouth image, and the ear scoring results of the ear image.
在本申请实施例中，对于不同的五官区域，五官图像有不同的评分标准。In the embodiment of the present application, different scoring criteria are used for facial feature images for different facial feature areas.
具体的，1)耳朵图像的评分标准与耳朵图像的清晰度相关。Specifically, 1) the scoring criteria for ear images are related to the clarity of the ear images.
在本申请实施例中，关于图像的清晰度值可以基于一种或一种以上清晰度的传统计算方法计算得到的清晰度值。例如，可以通过但不限于Brenner梯度函数、Laplacian梯度函数、SMD(灰度方差)函数、无参考图像评价指标(NIQE)、Brisque算法中的一项或一项以上传统方法计算得到清晰度值，并且，可以将清晰度值直接作为图像的清晰度评分结果，或者也可以将清晰度值转换为百分制或等级制的结果作为图像的清晰度评分结果，本文对此不做限定。In the embodiment of the present application, the clarity value of the image can be a clarity value calculated based on one or more traditional clarity calculation methods. For example, the clarity value can be calculated by one or more traditional methods including but not limited to Brenner gradient function, Laplacian gradient function, SMD (grayscale variance) function, non-reference image evaluation index (NIQE), Brisque algorithm, and the clarity value can be directly used as the clarity scoring result of the image, or the clarity value can be converted into a percentage or grade system as the clarity scoring result of the image, which is not limited in this document.
可理解的，计算图像的清晰度值还可以采用深度学习方法，本文对此不做限定。It is understandable that the clarity value of an image may also be calculated using a deep learning method, which is not limited in this article.
2)眉毛图像的评分标准与眉毛图像的清晰度以及眉毛图像的色彩对比度相关。2) The scoring criteria for eyebrow images are related to the clarity of the eyebrow images and the color contrast of the eyebrow images.
示例性的，眉毛图像的评分结果包含眉毛图像的清晰度值以及眉毛图像的色彩对比度信息的相关参数(例如眉毛区域图像像素点的均值、方差等参数)。Exemplarily, the scoring result of the eyebrow image includes the clarity value of the eyebrow image and related parameters of the color contrast information of the eyebrow image (such as the mean value, variance and other parameters of the pixel points of the eyebrow area image).
3)鼻子图像的评分标准与鼻子图像的清晰度以及高光信息相关。3) The scoring criteria for nose images are related to the clarity and highlight information of the nose images.
示例性的，鼻子图像的评分结果包含鼻子图像的清晰度值以及鼻子图像的高光信息相关参数。例如可以基于鼻子图像的RGB域或HSV域的色彩对比度信息提取用于表示鼻子图像的高光情况的特征，例如色彩均值方差等，一般认为对应区域的像素点的色彩对比度需要存在一定的差异，该区域图像的立体度更好。Exemplarily, the scoring result of the nose image includes the clarity value of the nose image and the parameters related to the highlight information of the nose image. For example, the features used to represent the highlight situation of the nose image, such as the color mean variance, can be extracted based on the color contrast information of the RGB domain or HSV domain of the nose image. It is generally believed that the color contrast of the pixels in the corresponding area needs to have a certain difference, and the three-dimensionality of the image in this area is better.
4)眼睛图像的评分标准与眼睛图像的清晰度以及眼睛睁开程度相关，眼睛睁开程度越大越好。4) The scoring criteria for eye images are related to the clarity of the eye images and the degree of eye openness. The greater the degree of eye openness, the better.
示例性的，眼睛图像的评分结果包含眼睛图像的清晰度值和眼睛睁开程度信息(眼睛睁开程度信息可以为虹膜面积或眼球面积)。Exemplarily, the scoring result of the eye image includes the clarity value of the eye image and the eye openness information (the eye openness information may be the iris area or the eyeball area).
5)嘴巴图像的评分标准与嘴唇图像的纹理以及嘴巴的张开程度相关。5) The scoring criteria for mouth images are related to the texture of the lip image and the degree of mouth opening.
示例性的，嘴巴图像的评分结果包含嘴唇图像的纹理值和嘴巴张开程度，关于嘴唇图像的纹理值的计算可以参照上述关于面部图像的纹理值的计算方式，在此不再详述。Exemplarily, the scoring result of the mouth image includes the texture value of the lip image and the degree of mouth opening. The calculation of the texture value of the lip image can refer to the above-mentioned calculation method of the texture value of the facial image, which will not be described in detail here.
在另外一些可能的实现方式中，电子设备还可以对第一图像进行头部姿态估计。具体为获取第一图像中人物的头部姿态旋转角(也称头部姿态欧拉角)，例如可以基于头部姿态估计(head pose estimation)算法，获取第一图像的头部姿态。In some other possible implementations, the electronic device may also perform head pose estimation on the first image, specifically obtaining the head pose rotation angle (also called head pose Euler angle) of the person in the first image, for example, the head pose of the first image may be obtained based on a head pose estimation algorithm.
S505，电子设备获取第一图像的人脸聚类标签。S505: The electronic device obtains a face clustering label of the first image.
示例性的，电子设备通过人物处理子系统30中人物处理模块的人脸聚类功能确定第一图像的人脸聚类标签。Exemplarily, the electronic device determines the face clustering label of the first image through the face clustering function of the character processing module in the character processing subsystem 30.
示例性的，基于人脸检测和人脸特征提取使用深度学习方法对第一图像进行人脸聚类，获得第一图像的人脸聚类标签。具体如何使用深度学习方法进行人脸聚类为本领域技术人员熟知的技术，在此不再详述。可理解的，也可以使用其他已知的人脸聚类算法对第一图像进行人脸聚类，本文对此不做限定。Exemplarily, based on face detection and face feature extraction, a deep learning method is used to perform face clustering on the first image to obtain a face clustering label of the first image. How to use the deep learning method to perform face clustering is a technique well known to those skilled in the art and will not be described in detail here. It is understandable that other known face clustering algorithms can also be used to perform face clustering on the first image, which is not limited herein.
可理解的，第一图像的人脸聚类标签用于标识该第一图像中的人脸图像的所属人物。示例性的，第一图像中包含人脸图像A，若该人脸图像A与已存储的人脸聚类标签中的标签X对应的人物匹配，则第一图像的人脸聚类标签即为该标签X。若该人脸图像A与已存储的人脸聚类标签中的任一个聚类标签对应的人物均不匹配，则电子设备为该第一图像分配新的人脸聚类标签。It is understandable that the face clustering label of the first image is used to identify the person to whom the face image in the first image belongs. Exemplarily, the first image contains face image A. If the face image A matches the person corresponding to label X in the stored face clustering labels, the face clustering label of the first image is the label X. If the face image A does not match the person corresponding to any cluster label in the stored face clustering labels, the electronic device assigns a new face clustering label to the first image.
在本申请中，电子设备在获取到第一图像的人脸聚类标签后，将其存储在本地存储空间中，也可以存储到云端(电子设备的云服务器端)，本文对此不做限定。In the present application, after the electronic device obtains the face clustering label of the first image, it stores it in a local storage space, and may also store it in the cloud (cloud server side of the electronic device), which is not limited in this article.
可理解的，若图像的相关数据(包括第一图像的相关数据，例如第一图像的待增强五官图像、待增强面部图像、背景图像、面部评分结果、五官评分结果、人脸本征信息、人脸聚类标签等)存储在电子设备的本地存储空间中，则电子设备在获取图像的相关数据时，可以直接根据检索需求读取本地存储空间中存储的相应数据；若存储在电子设备的云服务器端，则电子设备在获取图像的相关数据时，需要向云端请求与检索需求对应的相应数据，需要电子设备需要接入可以与云服务器进行数据通信的网络。It is understandable that if the relevant data of the image (including the relevant data of the first image, such as the facial features image to be enhanced, the face image to be enhanced, the background image, the facial scoring results, the facial features scoring results, the facial intrinsic information, the facial clustering labels, etc. of the first image) is stored in the local storage space of the electronic device, then when the electronic device obtains the relevant data of the image, it can directly read the corresponding data stored in the local storage space according to the retrieval requirements; if it is stored on the cloud server side of the electronic device, when the electronic device obtains the relevant data of the image, it needs to request the corresponding data corresponding to the retrieval requirements from the cloud, and the electronic device needs to access a network that can communicate data with the cloud server.
S506，电子设备确定第一图像是否满足增强条件。S506: The electronic device determines whether the first image meets an enhancement condition.
上述增强条件包括但不限于以下至少一项：人像模糊、人脸纹理细节缺失、人脸区域图像噪声高、人脸区域光照不足、人脸区域光照过亮。若电子设备确定第一图像满足该增强条件中的其中一项条件，则第一图像满足增强条件。The enhancement conditions include but are not limited to at least one of the following: blurred portrait, missing facial texture details, high image noise in the facial area, insufficient lighting in the facial area, and excessive lighting in the facial area. If the electronic device determines that the first image meets one of the enhancement conditions, the first image meets the enhancement condition.
可理解的，造成人物图像的人像模糊、人脸纹理细节缺失、人脸区域图像噪声高、人脸区域光照不足、人脸区域光照过亮的场景，可以包括但不限于暗光场景、欠曝或过曝场景、数字变焦、对焦不准、拍照抖动中的一项或多项。It is understandable that the scenes that cause blurred portraits, missing facial texture details, high image noise in the face area, insufficient lighting in the face area, and excessive lighting in the face area may include but are not limited to one or more of dark scenes, underexposure or overexposure scenes, digital zoom, inaccurate focus, and camera shake.
示例性的，可以通过N(N大于或等于1)种清晰度计算方法得到第一图像的N项清晰度值，基于该N项清晰度值确定第一图像是否满足人像模糊的增强条件。具体的，该N项清晰度值分别对应地满足N个模糊度阈值的情况下，确定该第一图像满足人像模糊条件。例如，该N中清晰度计算方法包括Brenner梯度函数、Laplacian梯度函数、SMD函数、NIQE三种，对应地N个模糊度阈值包括阈值A(与Brenner对应)、阈值B(与Laplacian对应)、阈值C(与SMD对应)、阈值D(与NIQE对应)，若第一图像针对该N个清晰度计算方法的N项清晰度值中Brenner梯度函数对应的清晰度值小于该阈值A、且Laplacian梯度函数对应的清晰度值小于阈值B、且SMD函数对应的清晰度值小于该阈值C、以及NIQE算法对应的清晰度值大于该阈值D，则该第一图像满足人像模糊的增强条件。Exemplarily, N clarity values of the first image can be obtained by N (N is greater than or equal to 1) clarity calculation methods, and whether the first image meets the enhancement condition of portrait blur can be determined based on the N clarity values. Specifically, when the N clarity values respectively meet N blur thresholds, it is determined that the first image meets the portrait blur condition. For example, the N clarity calculation methods include three types: Brenner gradient function, Laplacian gradient function, SMD function, and NIQE, and the corresponding N blur thresholds include threshold A (corresponding to Brenner), threshold B (corresponding to Laplacian), threshold C (corresponding to SMD), and threshold D (corresponding to NIQE). If the clarity value corresponding to the Brenner gradient function in the N clarity values of the N clarity calculation methods of the first image is less than the threshold A, and the clarity value corresponding to the Laplacian gradient function is less than the threshold B, and the clarity value corresponding to the SMD function is less than the threshold C, and the clarity value corresponding to the NIQE algorithm is greater than the threshold D, then the first image meets the enhancement condition of portrait blur.
示例性的，通过计算第一图像的人脸区域的纹理特征值确定第一图像是否满足人脸纹理细节缺失的增强条件。例如，可以基于待增强面部图像的LBP模式、灰度共生矩阵、分形维数、Tumera中的一项或多项数据计算第一图像中的目标块的纹理值，该目标块为第一图像的面部区域(包括脸颊区域、额头区域)中预设面部大小的图像。若任意一个目标块的纹理值满足用于表示纹理细节缺失的纹理值阈值，则第一图像满足人脸纹理细节缺失的增强条件。Exemplarily, whether the first image satisfies the enhancement condition of missing facial texture details is determined by calculating the texture feature value of the facial region of the first image. For example, the texture value of the target block in the first image can be calculated based on one or more data of the LBP mode, gray level co-occurrence matrix, fractal dimension, and Tumera of the facial image to be enhanced, and the target block is an image of a preset facial size in the facial region (including the cheek region and the forehead region) of the first image. If the texture value of any target block satisfies the texture value threshold used to indicate the missing texture details, the first image satisfies the enhancement condition of missing facial texture details.
示例性的，基于图像噪声计算方法确定第一图像是否满足人脸区域图像噪声高的增强条件。Exemplarily, it is determined whether the first image satisfies the enhancement condition that the image noise in the face area is high based on the image noise calculation method.
示例性的，电子设备可以对第一图像的人脸图像在RGB域或HSV域上进行亮度估计，若第一图像的人脸的亮度估计小于第一预设亮度阈值，则确定第一图像的人脸区域光照不足，当第一图像的人脸亮度估计大于第二预设亮度阈值，则确定第一图像的人脸区域光照过亮。或者，电子设备也可以将第一图像的人脸区域的RGB值转为灰度值，基于人脸区域图像的灰度值确定第一图像是否满足人脸区域光照过亮的增强条件。或者，电子设备也可以基于第一图像的人脸区域的HSV颜色模型，确定第一图像是否满足人脸区域光照过亮的增强条件。或者，电子设备还可以基于其他统计方法确定第一图像是否属于满足人脸区域光照不足或人脸光照过亮的增强条件，本文对此不做限定。Exemplarily, the electronic device may perform brightness estimation on the face image of the first image in the RGB domain or the HSV domain. If the brightness estimation of the face of the first image is less than the first preset brightness threshold, it is determined that the face area of the first image is insufficiently illuminated. When the brightness estimation of the face of the first image is greater than the second preset brightness threshold, it is determined that the face area of the first image is too bright. Alternatively, the electronic device may also convert the RGB value of the face area of the first image into a grayscale value, and determine whether the first image meets the enhancement condition of excessive brightness of the face area based on the grayscale value of the face area image. Alternatively, the electronic device may also determine whether the first image meets the enhancement condition of excessive brightness of the face area based on the HSV color model of the face area of the first image. Alternatively, the electronic device may also determine whether the first image meets the enhancement condition of insufficient brightness of the face area or excessive brightness of the face based on other statistical methods, which is not limited in this document.
或者，还可以基于传统的统计方法，例如通过泊松融合方法对第一图像的灰度图像进行统计或对第一图像在HSV域中的V明亮度维度进行统计，确定第一图像是否存在过曝或欠曝现象，以及具体的过曝或欠曝区域，以确定第一图像是否满足人脸区域光照不足或光照过亮的增强条件。Alternatively, it is also possible to determine whether the first image is overexposed or underexposed, as well as the specific overexposed or underexposed areas, based on traditional statistical methods, such as performing statistics on the grayscale image of the first image or performing statistics on the V brightness dimension of the first image in the HSV domain through a Poisson fusion method, so as to determine whether the first image meets the enhancement conditions for insufficient lighting or excessive lighting in the face area.
可理解的，在一些场景下，例如暗光下光照少的场景、欠曝过曝场景、变焦对焦不准场景、或拍照抖动场景，手机拍照得到的人像图像噪声普遍较高，脸部细节缺失，图像画质清晰度也普遍较差，从而需要进行图像增强，改善图像质量。Understandably, in some scenarios, such as low-light scenes, underexposure or overexposure scenes, inaccurate zoom focus scenes, or camera shaking scenes, the portrait images taken with mobile phones generally have high noise, missing facial details, and generally poor image clarity, thus requiring image enhancement to improve image quality.
在本申请中，上述步骤S503、S505、S506可以并行执行，也可以先后执行且其先后顺序不做限定。In the present application, the above steps S503, S505, and S506 can be executed in parallel or in sequence, and the order of execution is not limited.
在一种可能的实现方式中，在确定第一图像不满足增强条件的情况下，将第一图像存储到图像数据库50中，并在电子设备的电量大于预设电量阈值或电子设备处于充电灭屏状态的情况下，对第一图像执行上述步骤S503至S505。In one possible implementation, when it is determined that the first image does not meet the enhancement conditions, the first image is stored in the image database 50, and when the power of the electronic device is greater than a preset power threshold or the electronic device is in a charging screen-off state, the above steps S503 to S505 are performed on the first image.
在另外一些可能的实现方式中，电子设备在确定第一图像不满足增强条件之后还可以进一步确定第一图像的面部评分和五官评分是否较高，若确定第一图像的面部评分和五官评分较高，例如面部评分和五官评分以百分制为准，若第一图像的面部评分和五官评分均高于70分，则可以认为该第一图像之后被作为优质图像的可能性很高，则电子设备还可以对第一图像进行人脸本征估计，并将人脸本征信息存储到图像数据库50中，以便于之后需要使用该第一图像作为优质图像、且需要使用到优质图像的人脸本征信息时，可以直接读取图像数据库50中存储的相关人脸本征信息，减少图像处理的响应时间，提高用户体验。In some other possible implementations, after determining that the first image does not meet the enhancement conditions, the electronic device may further determine whether the facial score and facial feature score of the first image are high. If it is determined that the facial score and facial feature score of the first image are high, for example, the facial score and facial feature score are based on a percentage system, if the facial score and facial feature score of the first image are both higher than 70 points, then it can be considered that there is a high possibility that the first image will be used as a high-quality image in the future. The electronic device can also perform facial intrinsic estimation on the first image and store the facial intrinsic information in the image database 50, so that when the first image needs to be used as a high-quality image in the future and the facial intrinsic information of the high-quality image is needed, the relevant facial intrinsic information stored in the image database 50 can be directly read, thereby reducing the response time of image processing and improving user experience.
电子设备在确定第一图像满足增强条件的情况下，执行步骤S507；在确定第一图像不满足增强条件的情况下，不对第一图像进行增强处理(在图5中以步骤S513示出)，也即不执行步骤S507至S512。When the electronic device determines that the first image meets the enhancement condition, it executes step S507; when it determines that the first image does not meet the enhancement condition, it does not enhance the first image (shown as step S513 in Figure 5), that is, it does not execute steps S507 to S512.
阶段3：电子设备检索第一优质五官图像并利用第一优质五官图像对待增强五官图像进行增强处理，检索优质面部图像并利用优质面部图像对待增强面部图像进行增强处理，得到增强肖像。Stage 3: The electronic device retrieves a first high-quality facial feature image and uses the first high-quality facial feature image to enhance the facial feature image to be enhanced, retrieves a high-quality facial image and uses the high-quality facial image to enhance the facial image to be enhanced, and obtains an enhanced portrait.
S507，电子设备获取与第一图像的待增强五官图像匹配的第一优质五官图像。S507, the electronic device obtains a first high-quality facial feature image that matches the facial feature image to be enhanced of the first image.
在本申请实施例中，图像数据库50中存储有历史图像的人脸聚类标签、五官图像、面部图像、肖像、背景图像、面部评分结果、五官评分结果、人脸本征信息。其中，该历史图像是指电子设备在获取到第一图像之前存储到图像数据库50中的人物图像。In the embodiment of the present application, the image database 50 stores face clustering labels, facial features images, facial images, portraits, background images, facial scoring results, facial features scoring results, and facial intrinsic information of historical images. The historical images refer to images of people stored in the image database 50 by the electronic device before the first image is acquired.
在本申请实施例中，第一优质五官图像包括第一眉毛图像、第一眼睛图像、第一鼻子图像、第一嘴巴图像、以及第一耳朵图像，且第一优质五官图像中的每个五官部位对应的图像可以来源于同一张图像或来源于不同图像。以下介绍电子设备具体如何选取与待增强五官图像匹配的第一优质五官图像中的各项五官部位对应的图像。以下为便于描述，将第一图像中的待增强五官图像对应的人脸聚类标签所对应的人物称为目标人物。In an embodiment of the present application, the first high-quality facial feature image includes a first eyebrow image, a first eye image, a first nose image, a first mouth image, and a first ear image, and the image corresponding to each facial feature part in the first high-quality facial feature image can be derived from the same image or from different images. The following describes how the electronic device specifically selects the image corresponding to each facial feature part in the first high-quality facial feature image that matches the facial feature image to be enhanced. For ease of description below, the person corresponding to the face clustering label corresponding to the facial feature image to be enhanced in the first image is referred to as the target person.
1)第一眉毛图像1) First eyebrow image
示例性的，电子设备确定眉毛图像集合以及确定该眉毛图像集合中眉毛评分结果最优的眉毛图像作为上述第一眉毛图像。其中，该眉毛图像集合中的任意一项眉毛图像来源于目标人物，且该眉毛图像集合中的任意一项眉毛图像的眉毛评分结果优于待增强五官图像中的眉毛图像，以及该眉毛图像集合中的任意一项眉毛图像的眉毛评分结果均满足对应的评分阈值，例如，该眉毛图像集合中的任意一项眉毛图像的眉毛评分结果为N项清晰度值和色彩对比度相关的参数，则该N项清晰度值均满足对应的预设清晰度阈值，且该色彩对比度反映的立体度信息较优。上述确定该眉毛图像集合中眉毛评分结果最优的眉毛图像作为上述第一眉毛图像具体可以包括：基于眉毛评分函数和相应的加权系数将该N项清晰度值和色彩对比度的相关参数计算眉毛综合评分结果，基于眉毛综合评分结果选取该眉毛图像集合中清晰度最优的眉毛图像作为该第一眉毛图像。本申请所描述的加权系数的具体取值根据具体情况和具体需求而定。Exemplarily, the electronic device determines an eyebrow image set and determines the eyebrow image with the best eyebrow scoring result in the eyebrow image set as the above-mentioned first eyebrow image. Among them, any eyebrow image in the eyebrow image set comes from the target person, and the eyebrow scoring result of any eyebrow image in the eyebrow image set is better than the eyebrow image in the facial features image to be enhanced, and the eyebrow scoring result of any eyebrow image in the eyebrow image set meets the corresponding scoring threshold. For example, the eyebrow scoring result of any eyebrow image in the eyebrow image set is N parameters related to clarity values and color contrast, then the N clarity values all meet the corresponding preset clarity threshold, and the stereoscopic information reflected by the color contrast is better. The above-mentioned determination of the eyebrow image with the best eyebrow scoring result in the eyebrow image set as the above-mentioned first eyebrow image may specifically include: calculating the eyebrow comprehensive scoring result based on the eyebrow scoring function and the corresponding weighting coefficient, and selecting the eyebrow image with the best clarity in the eyebrow image set as the first eyebrow image based on the eyebrow comprehensive scoring result. The specific values of the weighting coefficients described in this application depend on the specific circumstances and specific requirements.
需要说明的是，该眉毛图像集合中的任意一项眉毛图像的眉毛评分结果优于待增强五官图像中的眉毛图像，可以是指该眉毛图像集合中的任意一项眉毛图像的N项清晰度值和色彩对比度相关的参数均对应地优于待增强五官图像中的眉毛图像。或者，也可以是指该眉毛图像集合中的任意一项眉毛图像的眉毛综合评分结果优于待增强五官图像中的眉毛综合评分结果。It should be noted that the eyebrow scoring result of any eyebrow image in the eyebrow image set is better than the eyebrow image in the facial feature image to be enhanced, which may mean that the N clarity values and color contrast-related parameters of any eyebrow image in the eyebrow image set are correspondingly better than the eyebrow image in the facial feature image to be enhanced. Alternatively, it may also mean that the eyebrow comprehensive scoring result of any eyebrow image in the eyebrow image set is better than the eyebrow comprehensive scoring result in the facial feature image to be enhanced.
2)第一眼睛图像2) First Eye Image
示例性的，电子设备先确定眼睛图像集合，再确定该眼睛图像集合中眼睛评分结果最优的眼睛图像作为上述第一眼睛图像。其中，该眼睛图像集合中的任意一项眼睛图像来源于目标人物，且眼睛图像集合中的任意一项眼睛图像的眼睛评分结果优于待增强五官图像中的眼睛图像，以及该眼睛图像集合中的任意一项眼睛图像的眼睛评分结果满足对应的评分阈值，例如该眼睛图像集合中的任意一项眼睛图像的眼睛评分结果包含N项清晰度和眼睛睁开程度，则该N项清晰度值均满足对应的预设清晰度阈值，且该眼睛睁开程度大于或等于预设睁开程度。上述确定该眼睛图像集合中眼睛评分结果最优的眼睛图像作为上述第一眼睛图像具体可以包括：基于眼睛评分函数和相应的加权系数计算该N项清晰度值和该眼睛睁开程度对应的眼睛综合评分结果，基于该眼睛综合评分结果选取该眼睛图像集合中得分最优的眼睛图像作为该第一眼睛图像。在另外一些可能的实现方式中，眼睛图像集合中的每一项眼睛图像还需要满足包含眼神光的条件。Exemplarily, the electronic device first determines an eye image set, and then determines the eye image with the best eye scoring result in the eye image set as the above-mentioned first eye image. Among them, any eye image in the eye image set comes from the target person, and the eye scoring result of any eye image in the eye image set is better than the eye image in the facial features image to be enhanced, and the eye scoring result of any eye image in the eye image set meets the corresponding scoring threshold. For example, the eye scoring result of any eye image in the eye image set includes N items of clarity and eye opening degree, then the N items of clarity values all meet the corresponding preset clarity threshold, and the eye opening degree is greater than or equal to the preset opening degree. The above-mentioned determination of the eye image with the best eye scoring result in the eye image set as the above-mentioned first eye image may specifically include: calculating the eye comprehensive scoring result corresponding to the N items of clarity values and the eye opening degree based on the eye scoring function and the corresponding weighting coefficient, and selecting the eye image with the best score in the eye image set as the first eye image based on the eye comprehensive scoring result. In some other possible implementations, each eye image in the eye image set also needs to meet the condition of containing eye light.
需要说明的是，该眼睛图像集合中的任意一项眼睛图像的眼睛评分结果优于待增强五官图像中的眼睛图像，可以是指该眼睛图像集合中的任意一项眼睛图像的N项清晰度值和眼睛睁开程度均对应地优于待增强五官图像中的眼睛图像的N项清晰度值以及眼睛睁开程度。或者，也可以是指该眼睛图像集合中的任意一项眼睛图像的眼睛综合评分结果优于待增强五官图像中的眼睛综合评分结果。It should be noted that the eye scoring result of any eye image in the eye image set is better than the eye image in the facial features image to be enhanced, which may mean that the N clarity values and eye openness of any eye image in the eye image set are correspondingly better than the N clarity values and eye openness of the eye image in the facial features image to be enhanced. Alternatively, it may also mean that the eye comprehensive scoring result of any eye image in the eye image set is better than the eye comprehensive scoring result in the facial features image to be enhanced.
3)第一鼻子图像3) First nose image
示例性的，电子设备先确定鼻子图像集合，再确定该鼻子图像集合中鼻子评分结果最优的鼻子图像作为上述第一鼻子图像。其中，该鼻子图像集合中的任意一项鼻子图像来源于目标人物，且该鼻子图像集合中的任意一项鼻子图像的眼睛评分结果优于待增强五官图像中的鼻子图像，以及该鼻子图像集合中的任意一项鼻子图像的鼻子评分结果满足对应的评分阈值，例如该鼻子图像集合中的任意一项鼻子图像的鼻子评分结果包含N项清晰度和高光信息的相关参数，则该N项清晰度值均满足对应的预设清晰度阈值，且高光信息的相关参数反映的立体度信息较优。上述确定该鼻子图像集合中鼻子评分结果最优的鼻子图像作为上述第一鼻子图像具体可以包括：基于鼻子评分函数和相应的加权系数计算该N项清晰度值和该高光信息的相关参数对应的鼻子综合评分结果，基于该鼻子综合评分结果选取该鼻子图像集合中得分最优的鼻子图像作为该第一鼻子图像。Exemplarily, the electronic device first determines a nose image set, and then determines the nose image with the best nose scoring result in the nose image set as the above-mentioned first nose image. Among them, any nose image in the nose image set comes from the target person, and the eye scoring result of any nose image in the nose image set is better than the nose image in the facial features image to be enhanced, and the nose scoring result of any nose image in the nose image set meets the corresponding scoring threshold. For example, the nose scoring result of any nose image in the nose image set includes N parameters related to clarity and highlight information, then the N clarity values all meet the corresponding preset clarity threshold, and the stereoscopic information reflected by the related parameters of the highlight information is better. The above-mentioned determination of the nose image with the best nose scoring result in the nose image set as the above-mentioned first nose image can specifically include: calculating the nose comprehensive scoring result corresponding to the N clarity values and the related parameters of the highlight information based on the nose scoring function and the corresponding weighting coefficient, and selecting the nose image with the best score in the nose image set as the first nose image based on the nose comprehensive scoring result.
需要说明的是，该鼻子图像集合中的任意一项鼻子图像的眼睛评分结果优于待增强五官图像中的鼻子图像，可以是指该眼睛图像集合中的任意一项眼睛图像的N项清晰度值和高光信息均对应地优于待增强五官图像中的眼睛图像。或者，也可以是指该眼睛图像集合中的任意一项眼睛图像的眼睛综合评分结果优于待增强五官图像中的眼睛综合评分结果。It should be noted that the eye scoring result of any nose image in the nose image set is better than the nose image in the facial features image to be enhanced, which may mean that the N clarity values and highlight information of any eye image in the eye image set are correspondingly better than the eye image in the facial features image to be enhanced. Alternatively, it may also mean that the eye comprehensive scoring result of any eye image in the eye image set is better than the eye comprehensive scoring result in the facial features image to be enhanced.
4)第一嘴巴图像4) First mouth image
示例性的，电子设备先确定嘴巴图像集合，再确定该嘴巴图像集合中嘴巴评分结果最优的嘴巴图像作为上述第一嘴巴图像。其中，该嘴巴图像集合中的任意一项嘴巴图像来源于目标人物，且该嘴巴图像集合中的任意一项嘴巴图像的嘴巴评分结果优于待增强五官图像中的嘴巴图像，以及该嘴巴图像集合中的每一项嘴巴图像的嘴巴张开程度小于预设张开程度。上述确定该嘴巴图像集合中嘴巴评分结果最优的嘴巴图像作为上述第一嘴巴图像具体可以包括：基于嘴巴评分函数和相应的加权系数计算该纹理值和该嘴巴张开程度信息对应的嘴巴综合评分结果，基于该嘴巴综合评分结果选取该嘴巴图像集合中得分最优的嘴巴图像作为该第一嘴巴图像。Exemplarily, the electronic device first determines a mouth image set, and then determines the mouth image with the best mouth score result in the mouth image set as the above-mentioned first mouth image. Among them, any mouth image in the mouth image set originates from the target person, and the mouth score result of any mouth image in the mouth image set is better than the mouth image in the facial features image to be enhanced, and the mouth opening degree of each mouth image in the mouth image set is less than a preset opening degree. The above-mentioned determination of the mouth image with the best mouth score result in the mouth image set as the above-mentioned first mouth image can specifically include: calculating the mouth comprehensive score result corresponding to the texture value and the mouth opening degree information based on the mouth score function and the corresponding weighting coefficient, and selecting the mouth image with the best score in the mouth image set as the first mouth image based on the mouth comprehensive score result.
需要说明的是，该嘴巴图像集合中的任意一项嘴巴图像的嘴巴评分结果优于待增强五官图像中的嘴巴图像，可以是指该嘴巴图像集合中的任意一项嘴巴图像的纹理值和嘴巴张开程度均对应地优于待增强五官图像中的嘴巴图像。或者，也可以是指该嘴巴图像集合中的任意一项嘴巴图像的嘴巴综合评分结果优于待增强五官图像中的嘴巴综合评分结果。It should be noted that the mouth score result of any mouth image in the mouth image set is better than the mouth image in the facial features image to be enhanced, which may mean that the texture value and mouth opening degree of any mouth image in the mouth image set are correspondingly better than the mouth image in the facial features image to be enhanced. Alternatively, it may also mean that the comprehensive mouth score result of any mouth image in the mouth image set is better than the comprehensive mouth score result in the facial features image to be enhanced.
5)第一耳朵图像5) First Ear Image
示例性的，电子设备先确定耳朵图像集合，再确定该耳朵图像集合中耳朵评分结果最优的耳朵图像作为上述第一耳朵图像。其中，该耳朵图像集合中的任意一项耳朵图像来源于目标人物，且该耳朵图像集合中的任意一项耳朵图像的耳朵评分结果优于待增强五官图像中的耳朵图像，以及该耳朵图像集合中的任意一项耳朵图像的耳朵评分结果满足对应的评分阈值，例如该耳朵图像集合中的任意一项耳朵图像的耳朵评分结果为N项清晰度值，则该N项清晰度值均满足对应的预设清晰度阈值。上述确定该耳朵图像集合中耳朵评分结果最优的耳朵图像作为上述第一耳朵图像具体可以包括：基于耳朵评分函数和相应的加权系数将耳朵图像的N项清晰度值综合为一个清晰度值，基于综合的清晰度值选取该耳朵图像集合中综合清晰度最优的耳朵图像作为该第一嘴巴图像。Exemplarily, the electronic device first determines an ear image set, and then determines the ear image with the best ear scoring result in the ear image set as the above-mentioned first ear image. Among them, any ear image in the ear image set originates from the target person, and the ear scoring result of any ear image in the ear image set is better than the ear image in the facial features image to be enhanced, and the ear scoring result of any ear image in the ear image set meets the corresponding scoring threshold. For example, the ear scoring result of any ear image in the ear image set is N clarity values, then the N clarity values all meet the corresponding preset clarity threshold. The above-mentioned determination of the ear image with the best ear scoring result in the ear image set as the above-mentioned first ear image can specifically include: combining the N clarity values of the ear image into one clarity value based on the ear scoring function and the corresponding weighting coefficient, and selecting the ear image with the best comprehensive clarity in the ear image set as the first mouth image based on the comprehensive clarity value.
需要说明的是，该耳朵图像集合中的任意一项耳朵图像的耳朵评分结果优于待增强五官图像中的耳朵图像，可以是指该耳朵图像集合中的任意一项耳朵图像的N项清晰度值均对应地优于待增强五官图像中的耳朵图像的N项清晰度值。或者，也可以是指该耳朵图像集合中的任意一项耳朵图像的耳朵综合评分结果优于待增强五官图像中的耳朵综合评分结果。It should be noted that the ear scoring result of any ear image in the ear image set is better than the ear image in the facial features image to be enhanced, which may mean that the N clarity values of any ear image in the ear image set are correspondingly better than the N clarity values of the ear image in the facial features image to be enhanced. Alternatively, it may also mean that the ear comprehensive scoring result of any ear image in the ear image set is better than the ear comprehensive scoring result in the facial features image to be enhanced.
S508，电子设备利用第一优质五官图像对待增强五官图像进行五官增强处理，获得增强后的五官图像。S508, the electronic device uses the first high-quality facial feature image to perform facial feature enhancement processing on the facial feature image to be enhanced to obtain an enhanced facial feature image.
也可以理解为将第一优质五官图像中包含的优质的五官特征作为引导对待增强面部图像进行增强处理，该优质的五官特征可以包括但不限于：清晰的画质、细腻的嘴唇纹理、眉毛对比度清晰(毛流感好)、清晰的耳朵、鼻子区域较好的几何立体感中的一项或多项。It can also be understood that the high-quality facial features contained in the first high-quality facial feature image are used as a guide to enhance the facial image to be enhanced. The high-quality facial features may include but are not limited to: one or more of: clear image quality, delicate lip texture, clear eyebrow contrast (good hair flow), clear ears, and good geometric three-dimensional sense of the nose area.
可选的，利用第一优质五官图像对待增强五官图像进行五官增强处理的方法可以包括但不限于传统方法或深度学习方法。Optionally, the method of performing facial feature enhancement processing on the facial feature image to be enhanced using the first high-quality facial feature image may include but is not limited to a traditional method or a deep learning method.
例如电子设备可以通过深度学习在隐编码层对待增强五官图像进行增强，具体的，利用训练好的五官增强网络模型对待增强五官图像和第一优质五官图像进行特征提取，基于提取到的特征信息进行特征拟合计算，以便于进行特征融合或特征迁移，使得将第一优质五官图像中的优质特征融合到待增强五官图像中，对待增强五官图像进行增强。For example, electronic devices can enhance the facial feature image to be enhanced in the hidden coding layer through deep learning. Specifically, the trained facial feature enhancement network model is used to extract features of the facial feature image to be enhanced and the first high-quality facial feature image, and feature fitting calculations are performed based on the extracted feature information to facilitate feature fusion or feature migration, so that the high-quality features in the first high-quality facial feature image are fused into the facial feature image to be enhanced, thereby enhancing the facial feature image to be enhanced.
示例性的，利用第一优质五官图像对待增强五官图像进行五官增强可以包括但不限于：嘴唇纹理细节的增强、眉毛对比度的增强(以使得增强眉毛区域的毛流感细节)、鼻子区域色彩对比度的增强(以使得提高鼻子区域对应图像的几何立体度)、耳朵清晰的增强、眼神光的光斑轮廓清晰度和光斑亮度的增强、眼睛虹膜色彩信息和虹膜纹理的增强。其中，眼睛虹膜色彩信息和虹膜纹理的增强具体包括但不限于：在虹膜信息库中查询与上述待增强五官图像的虹膜颜色相似度较高的参考虹膜图像，基于该参考虹膜图像的虹膜纹理信息和色彩信息，对该待增强五官图像的虹膜纹理和虹膜色彩进行增强处理，其中该虹膜信息库中可以包含两个或两个以上不同虹膜颜色的清晰且纹理细节优质的虹膜样板图像。Exemplarily, the facial enhancement of the facial features image to be enhanced using the first high-quality facial features image may include but is not limited to: enhancing the lip texture details, enhancing the eyebrow contrast (so as to enhance the hair flow details of the eyebrow area), enhancing the nose area color contrast (so as to improve the geometric three-dimensionality of the image corresponding to the nose area), enhancing the ear clarity, enhancing the spot contour clarity and spot brightness of the eye light, and enhancing the eye iris color information and iris texture. Among them, the enhancement of the eye iris color information and iris texture specifically includes but is not limited to: querying the iris information database for a reference iris image with a high degree of similarity to the iris color of the facial features image to be enhanced, and enhancing the iris texture and iris color of the facial features image to be enhanced based on the iris texture information and color information of the reference iris image, wherein the iris information database may contain two or more clear iris sample images with high-quality texture details of different iris colors.
具体的，可以基于五官图像训练集和机器学习原理训练得到的上述五官增强模型，具体的可以是基于有监督学习方法或无监督学习方法训练得到，其中，该五官增强模型的训练目标是，当向模型中输入原始图像(或称原始五官图像)和该原始图像对应的优质图像(或称优质五官图像)后，提取两张图像的特征信息进行特征融合或特征迁移计算，输出的图像的画质清晰度更高、嘴唇区域图像纹理更多、眉毛区域图像的毛流感更佳(例如对比度更明显)、鼻子几何立体度更好(例如体现鼻子几何阴影信息)、眼神光的光斑轮廓更清晰和光斑亮度更好、眼睛虹膜色彩和虹膜纹理细节更好。Specifically, the facial feature enhancement model can be obtained based on a facial feature image training set and machine learning principles, and can be obtained based on a supervised learning method or an unsupervised learning method. The training goal of the facial feature enhancement model is to extract the feature information of the two images for feature fusion or feature migration calculation after inputting the original image (or the original facial feature image) and the high-quality image (or the high-quality facial feature image) corresponding to the original image into the model. The output image has higher picture clarity, more image texture in the lip area, better hair flow in the eyebrow area image (for example, more obvious contrast), better geometric three-dimensionality of the nose (for example, reflecting the geometric shadow information of the nose), clearer spot contour and better spot brightness of the eye light, and better details of the iris color and iris texture.
在本申请实施例中，五官增强模型可以是集成网络也可以是多个子网络的集合，该集成网络将基于第一眼睛图像、第一眉毛图像、第一嘴巴图像、第一耳朵图像、以及第一鼻子图像对待增强五官图像进行图像增强的功能集成到同一个模型；示例性的，该多个子网络包括基于第一眼睛图像对待增强五官图像中的眼睛图像进行增强处理的子网络、基于第一鼻子图像对待增强五官图像中的鼻子图像进行增强处理的子网络、基于第一嘴巴图像对待增强五官图像中的嘴巴图像进行增强处理的子网络、基于第一耳朵图像对待增强五官图像中的耳朵图像进行增强处理的功能对应的子网络、基于第一鼻子图像对待增强五官图像中的鼻子图像进行增强处理的功能对应的子网络。In an embodiment of the present application, the facial feature enhancement model can be an integrated network or a collection of multiple sub-networks, which integrates the function of performing image enhancement on the facial feature image to be enhanced based on the first eye image, the first eyebrow image, the first mouth image, the first ear image, and the first nose image into the same model; exemplarily, the multiple sub-networks include a sub-network for enhancing the eye image in the facial feature image to be enhanced based on the first eye image, a sub-network for enhancing the nose image in the facial feature image to be enhanced based on the first nose image, a sub-network for enhancing the mouth image in the facial feature image to be enhanced based on the first mouth image, a sub-network corresponding to the function of enhancing the ear image in the facial feature image to be enhanced based on the first ear image, and a sub-network corresponding to the function of enhancing the nose image in the facial feature image to be enhanced based on the first nose image.
示例性的，基于有监督学习方法训练得到上述五官增强模型，上述五官图像训练集至少包括两组样本数据，每一组样本数据包含样本图像1、样本图像2、以及样本图像3，样本图像1与样本图像3属于相同肖像图像的两张不同图像质量的五官图像，其中样本图像3的五官图像质量优于样本图像1，样本图像2与样本图像1对应同一人物(例如可以是相同人物的头部姿态和/或脸部微表情有所差异的两张不同图像)、且该样本图像2的五官图像质量优于样本图像1的五官图像质量，该样本图像1和样本图像2可以作为五官增强模型中原始五官图像的输入，样本图像3可以作为五官增强模型的输出图像的对比图像(也可以理解为约束图像)，该样本图像3可以用于指导五官增强模型中卷积层的损失函数和梯度回传。Exemplarily, the facial feature enhancement model is obtained by training based on a supervised learning method, and the facial feature image training set includes at least two groups of sample data, each group of sample data includes sample image 1, sample image 2, and sample image 3, sample image 1 and sample image 3 are two facial feature images of the same portrait image with different image qualities, wherein the facial feature image quality of sample image 3 is better than that of sample image 1, sample image 2 and sample image 1 correspond to the same person (for example, they can be two different images of the same person with different head postures and/or facial micro-expressions), and the facial feature image quality of sample image 2 is better than that of sample image 1, the sample image 1 and sample image 2 can be used as inputs of the original facial feature images in the facial feature enhancement model, sample image 3 can be used as a comparison image of the output image of the facial feature enhancement model (can also be understood as a constraint image), and the sample image 3 can be used to guide the loss function and gradient return of the convolutional layer in the facial feature enhancement model.
例如，上述样本图像1可以是对上述样本图像3进行降质操作得到的、包含暗光场景下图像质量较差的、存在欠曝或过曝区域的质量较差的五官图像，五官评分较低的五官图像。For example, the sample image 1 may be a facial feature image obtained by degrading the sample image 3, including a facial feature image with poor image quality in a dark light scene, an image with underexposed or overexposed areas, and a facial feature image with a low facial feature score.
示例性的，基于无监督学习方法训练得到上述五官增强模型，上述五官图像训练集至少包括两组样本数据，每一组样本数据包含样本图像4和样本图像5，该样本图像4和样本图像5作为五官增强模型中原始五官图像的输入，该五官增强模型包括生成模型和对抗模型，其中生成模型用于以样本图像5作为指导图像生成与样本图像4对应的增强处理后的输出图像，对抗模型为预训练好的用于评判输出图像是否符合增强效果的评判网络，样本图像4与样本图像5对应同一人物且该样本图像5的五官图像质量优于样本图像4的五官图像质量。Exemplarily, the facial feature enhancement model is obtained by training based on an unsupervised learning method, and the facial feature image training set includes at least two groups of sample data, each group of sample data includes sample image 4 and sample image 5, and the sample image 4 and sample image 5 are used as inputs of the original facial feature images in the facial feature enhancement model. The facial feature enhancement model includes a generative model and an adversarial model, wherein the generative model is used to generate an enhanced output image corresponding to the sample image 4 using the sample image 5 as a guide image, and the adversarial model is a pre-trained judgment network for judging whether the output image meets the enhancement effect, and the sample image 4 and the sample image 5 correspond to the same person and the facial feature image quality of the sample image 5 is better than the facial feature image quality of the sample image 4.
在一种可能的实现方式中，上述五官增强网络模型可以对电子设备中存储的所有历史人物图像的五官图像进行特征编码，并存储五官编码数据，当电子设备调用该五官增强网络模型对待增强五官图像进行增强时，五官增强网络模型可以直接查询与该待增强五官图像对应的五官编码数据，以及筛选比该待增强五官图像更优质的历史人物图像的特征编码数据，利用优质的五官编码数据和该待增强五官图像的五官编码数据进行特征融合处理，以对该待增强五官图像进行增强处理。也就是说，该五官增强网络模型可以达到上述步骤S507的功能效果，这种情况下，电子设备可以不再执行上述步骤S507。In a possible implementation, the facial features enhancement network model can perform feature encoding on the facial features images of all historical figures stored in the electronic device, and store the facial features encoding data. When the electronic device calls the facial features enhancement network model to enhance the facial features image to be enhanced, the facial features enhancement network model can directly query the facial features encoding data corresponding to the facial features image to be enhanced, and filter the feature encoding data of the historical figures image with better quality than the facial features image to be enhanced, and perform feature fusion processing using the high-quality facial features encoding data and the facial features encoding data of the facial features image to be enhanced, so as to enhance the facial features image to be enhanced. In other words, the facial features enhancement network model can achieve the functional effect of the above step S507. In this case, the electronic device may no longer execute the above step S507.
示例性的，利用传统方法对待增强五官图像进行增强是指，可以利用泊松函数、拉普拉斯函数，对第一优质五官图像中的像素值进行分析统计，提取图像的特征，例如提取的特征包括但不限于嘴唇的纹理特征(例如可以基于嘴唇图像的LBP、灰度共生矩阵、分形维数等计算嘴唇纹理特征)、眉毛对比度特征、鼻子几何立体度特征，获得图像质量好的该第一优质五官图像中像素值的分布情况，以及对待增强五官图像的像素值进行分析，并基于该第一优质五官图像中像素值的分布情况指导修正该待增强五官图像中的像素值，以使得处理得到的图像的画质清晰度更高、嘴唇区域图像纹理更多、眉毛区域图像的毛流感更佳、鼻子几何立体度更好、眼神光的光斑轮廓更清晰和光斑亮度更好、眼睛虹膜色彩和虹膜纹理细节更好。Exemplarily, using traditional methods to enhance the facial features image to be enhanced means that the Poisson function and the Laplace function can be used to analyze and count the pixel values in the first high-quality facial features image to extract image features, such as, but not limited to, lip texture features (for example, lip texture features can be calculated based on the LBP, gray-level co-occurrence matrix, fractal dimension, etc. of the lip image), eyebrow contrast features, and nose geometric three-dimensional features, to obtain the distribution of pixel values in the first high-quality facial features image with good image quality, and to analyze the pixel values of the facial features image to be enhanced, and to guide the correction of the pixel values in the facial features image to be enhanced based on the distribution of pixel values in the first high-quality facial features image, so that the processed image has higher picture clarity, more image texture in the lip area, better hair flow in the eyebrow area, better nose geometric three-dimensionality, clearer spot contour and better spot brightness of the eye light, and better iris color and iris texture details.
可理解的，第一优质五官图像的评分较优也即清晰度越好，也可以理解为，第一优质五官图像是在光照环境比较好的条件下拍的，而光照环境较好的条件下拍得的五官必然立体感更强、眉毛的细节更清晰、鼻子更清晰高挺、嘴唇细节更清晰、五官的光影感更真实，从而利用第一优质五官图像对待增强图像进行增强，可以达到很好的增强效果。It is understandable that the first high-quality facial features image has a higher score, that is, the better the clarity. It can also be understood that the first high-quality facial features image is taken under conditions of relatively good lighting environment, and the facial features taken under conditions of relatively good lighting environment must have a stronger three-dimensional sense, clearer eyebrow details, clearer and taller nose, clearer lip details, and more realistic light and shadow of the facial features. Therefore, using the first high-quality facial features image to enhance the image to be enhanced can achieve a good enhancement effect.
S509，电子设备获取与第一图像的待增强面部图像匹配的优质面部图像。S509: The electronic device obtains a high-quality facial image that matches the facial image to be enhanced in the first image.
示例性的，电子设备获取上述优质面部图像有以下两种可选的实现方式：Exemplarily, there are two optional implementations for the electronic device to obtain the above-mentioned high-quality facial image:
方式1：电子设备基于上述待增强面部图像的人脸聚类标签、待增强面部图像的面部评分结果、以及图像数据库50，获取与该待增强面部图像匹配的优质面部图像。Mode 1: The electronic device obtains a high-quality facial image matching the facial image to be enhanced based on the face clustering label of the facial image to be enhanced, the facial scoring result of the facial image to be enhanced, and the image database 50 .
具体的，电子设备基于图像数据库50、第一图像的人脸聚类标签获取面部图像集合A，该面部图像集合A中的每一个人物图像的人脸聚类标签包含于第一图像的人脸聚类标签、且每一个人物图像的面部评分结果均优于第一图像的面部评分结果，以及，电子设备确定该面部图像集合A中的评分最优的面部图像作为上述优质面部图像。示例性的，面部图像的评分结果为面部纹理值，面部纹理值的具体计算方法参照步骤S504中的相关说明，相应地面部图像集合A中的最优评分是指面部评分集合A中面部纹理值最优的历史图像。Specifically, the electronic device obtains a facial image set A based on the image database 50 and the facial clustering label of the first image, wherein the facial clustering label of each character image in the facial image set A is included in the facial clustering label of the first image, and the facial scoring result of each character image is better than the facial scoring result of the first image, and the electronic device determines the facial image with the best score in the facial image set A as the above-mentioned high-quality facial image. Exemplarily, the scoring result of the facial image is a facial texture value, and the specific calculation method of the facial texture value refers to the relevant description in step S504, and accordingly, the best score in the facial image set A refers to the historical image with the best facial texture value in the facial scoring set A.
方式2：电子设备基于第一图像的人脸聚类标签、第一图像的面部评分结果、第一图像的头部姿态、以及图像数据库50，获取与第一图像匹配的优质面部图像。Method 2: The electronic device obtains a high-quality facial image matching the first image based on the face clustering label of the first image, the facial scoring result of the first image, the head posture of the first image, and the image database 50.
具体的，电子设备基于图像数据库50、第一图像的人脸聚类标签获取上述面部评分集合A，基于评分X(评分X为面部评分集合A中的每一个评分)对应的历史图像的头部姿态、第一图像的头部姿态、第一图像的面部评分结果、以及该评分X，确定该评分X对应的历史图像的面部综合评分结果，得到综合面部评分集合B，确定该综合面部评分集合B中的最高评分对应的历史图像的面部图像作为上述优质面部图像。示例性的，可以将第一图像的头部姿态与评分X对应的历史图像的头部姿态的差值、和评分X的加权和作为该历史图像的面部综合评分结果。具体的加权数可以根据实际情况调整，本文对此不做限定。Specifically, the electronic device obtains the above-mentioned facial score set A based on the image database 50 and the face clustering label of the first image, and determines the facial comprehensive score result of the historical image corresponding to the score X based on the head posture of the historical image corresponding to the score X (the score X is each score in the facial score set A), the head posture of the first image, the facial score result of the first image, and the score X, and obtains the comprehensive facial score set B, and determines the facial image of the historical image corresponding to the highest score in the comprehensive facial score set B as the above-mentioned high-quality facial image. Exemplarily, the difference between the head posture of the first image and the head posture of the historical image corresponding to the score X, and the weighted sum of the score X can be used as the facial comprehensive score result of the historical image. The specific weighting number can be adjusted according to the actual situation, and this document does not limit this.
在本申请实施例中，上述第一优质五官图像和优质面部图像还满足来源于未执行过图像增强处理的人物图像的条件。In the embodiment of the present application, the above-mentioned first high-quality facial features image and high-quality facial image also meet the condition that they are derived from human images that have not been subjected to image enhancement processing.
S510，电子设备利用优质面部图像对待增强面部图像进行面部增强处理，获得增强后的面部图像。S510, the electronic device performs facial enhancement processing on the facial image to be enhanced using the high-quality facial image to obtain an enhanced facial image.
也可以理解为将优质面部图像中包含的优质的面部特征作为引导对待增强面部图像进行增强处理，该优质的面部特征可以包括但不限于：清晰且细腻的皮肤纹理、自然的色调。It can also be understood that the high-quality facial features contained in the high-quality facial image are used as a guide to enhance the facial image to be enhanced. The high-quality facial features may include but are not limited to: clear and delicate skin texture, and natural color tone.
可选的，可以基于传统方法或深度学习方法利用优质面部图像对待增强面部图像进行面部增强处理，或者也可以基于OpenCV图像处理方法等其他方法利用优质面部图像对待增强面部图像进行面部增强处理，本文对此不做限定。Optionally, facial enhancement processing may be performed on the facial image to be enhanced using high-quality facial images based on traditional methods or deep learning methods, or facial enhancement processing may be performed on the facial image to be enhanced using high-quality facial images based on other methods such as OpenCV image processing methods, and this document does not limit this.
利用优质面部图像对待增强面部图像进行面部增强处理，可以包括但不限于：基于‘有指导人脸增强’的原理对待增强面部图像进行面部增强，具体的，利用训练好的面部增强网络模型对待增强面部图像和优质面部图像进行特征提取，基于提取到的特征信息进行特征拟合计算(也可以理解为特征融合或特征迁移)，使得将优质面部图像中的优质特征融合到待增强面部图像中。Performing facial enhancement processing on the facial image to be enhanced using the high-quality facial image may include but is not limited to: performing facial enhancement on the facial image to be enhanced based on the principle of ‘guided face enhancement’, specifically, using a trained facial enhancement network model to extract features from the facial image to be enhanced and the high-quality facial image, performing feature fitting calculation (which can also be understood as feature fusion or feature migration) based on the extracted feature information, so that high-quality features in the high-quality facial image are fused into the facial image to be enhanced.
具体的，可以基于面部图像训练集和有监督学习方法或无监督学习方法的机器学习原理训练得到的上述面部增强模型，其中，该面部增强模型的训练目标是，当向模型中输入原始图像(或称原始面部图像)和该原始图像对应的优质图像(或称优质面部图像)后，提取两张图像的特征信息进行特征融合或特征迁移计算，输出画质清晰度更高、脸部纹理细节更多的增强图像。Specifically, the above-mentioned facial enhancement model can be obtained by training based on a facial image training set and a supervised learning method or an unsupervised learning method based on machine learning principles, wherein the training goal of the facial enhancement model is to extract the feature information of the two images for feature fusion or feature migration calculation after inputting the original image (or original facial image) and the high-quality image (or high-quality facial image) corresponding to the original image into the model, and output an enhanced image with higher picture quality clarity and more facial texture details.
示例性的，基于有监督学习方法训练得到上述面部增强模型，上述面部图像训练集至少包括两组样本数据，每一组样本数据包含样本图像A，样本图像B、以及样本图像C，样本图像A与样本图像C属于相同肖像的两张不同图像质量的面部图像，其中样本图像C的面部图像质量优于样本图像A；样本图像B与样本图像A对应同一人物(头部姿态、脸部微表情有所差异的两张不同图像)、且该样本图像B的面部图像质量优于样本图像A的面部图像质量，该样本图像A和样本图像B可以作为五官增强模型中原始五官图像的输入，样本图像C可以作为五官增强模型的输出图像的对比图像(也可以理解为约束图像)，该样本图像C可以用于指导五官增强模型中卷积层的损失函数和梯度回传。Exemplarily, the above-mentioned facial enhancement model is obtained by training based on a supervised learning method, and the above-mentioned facial image training set includes at least two groups of sample data, each group of sample data includes sample image A, sample image B, and sample image C, sample image A and sample image C are two facial images of the same portrait with different image qualities, wherein the facial image quality of sample image C is better than that of sample image A; sample image B and sample image A correspond to the same person (two different images with different head postures and facial micro-expressions), and the facial image quality of sample image B is better than that of sample image A, the sample image A and sample image B can be used as inputs of original facial features images in the facial features enhancement model, sample image C can be used as a comparison image of the output image of the facial features enhancement model (can also be understood as a constraint image), and the sample image C can be used to guide the loss function and gradient return of the convolutional layer in the facial features enhancement model.
例如，该样本图像A可以是对该样本图像C进行降质操作(也即降低图像质量)得到的、包含暗光场景下图像质量较差的、存在欠曝或过曝区域的质量较差的面部图像，面部评分较低的面部图像。For example, the sample image A may be a facial image obtained by downgrading the sample image C (i.e., reducing the image quality), including poor image quality in dark light scenes, poor quality facial images with underexposed or overexposed areas, and a facial image with a low facial score.
基于传统方法利用优质面部图像对待增强面部图像进行面部增强处理，可以包括但不限于：利用泊松函数、拉普拉斯函数，对优质面部图像中的像素值进行分析统计，提取图像特征，例如提取的特征包括但不限于面部纹理特征(例如可以基于面部图像的LBP、灰度共生矩阵分形维数等计算面部纹理特征)，以获得图像质量好的该优质面部图像中像素值的分布情况，以及对待增强面部图像的像素值进行分析，并基于该优质面部图像中像素值的分布情况指导修正该待增强面部图像中的像素值。Performing facial enhancement processing on the facial image to be enhanced using a high-quality facial image based on traditional methods may include but is not limited to: using a Poisson function or a Laplace function to analyze and count pixel values in the high-quality facial image, extracting image features, such as extracting features including but not limited to facial texture features (for example, facial texture features may be calculated based on LBP of the facial image, gray-level co-occurrence matrix fractal dimension, etc.), so as to obtain the distribution of pixel values in the high-quality facial image with good image quality, and analyzing the pixel values of the facial image to be enhanced, and guiding the correction of pixel values in the facial image to be enhanced based on the distribution of pixel values in the high-quality facial image.
可理解的，将优质面部图像中细腻、色调自然的皮肤纹理迁移或融合到待增强图像中，从而可以提升待增强面部图像的面部肤质清晰度、纹理细腻程度、人像真实感。Understandably, the delicate and natural-toned skin texture in the high-quality facial image is migrated or fused into the image to be enhanced, thereby improving the facial skin clarity, texture delicacy, and portrait realism of the facial image to be enhanced.
例如，当待增强图像的光照属于暗光场景的情况下，采用上述优质的面部特征作为引导对该待增强面部图像进行增强处理，增强后的面部图像其皮肤纹理会更细腻、色调更自然，一方面，增强后的面部图像包含细腻、色调自然的皮肤纹理，可以解决暗光场景下人像噪点高、细节涂抹现象严重的问题；另外一方面，优质面部图像的光照充足，增强后的面部图像中包含的皮肤纹理对应的像素点也是光照充足的，从而还可以改善暗光场景下光照不足、导致图像清晰度欠佳、图像亮度较低的问题。For example, when the lighting of the image to be enhanced belongs to a dark light scene, the above-mentioned high-quality facial features are used as a guide to enhance the facial image to be enhanced. The skin texture of the enhanced facial image will be more delicate and the tone will be more natural. On the one hand, the enhanced facial image contains delicate and natural-toned skin texture, which can solve the problems of high portrait noise and serious detail smearing in dark light scenes; on the other hand, the high-quality facial image has sufficient lighting, and the pixels corresponding to the skin texture contained in the enhanced facial image are also well-lit, which can also improve the problem of insufficient lighting in dark light scenes, resulting in poor image clarity and low image brightness.
例如，当待增强图像存在欠曝或过曝区域的情况下，采用上述优质的面部特征作为引导对该待增强面部图像进行增强处理，增强后的面部图像中与待增强面部图像的欠曝或过曝区域对应的区域，填补了相应的细腻、色调自然的皮肤纹理细节，从而可以改善欠曝或过曝场景下人像细节缺失的问题。For example, when there are underexposed or overexposed areas in the image to be enhanced, the above-mentioned high-quality facial features are used as a guide to enhance the facial image to be enhanced. The area in the enhanced facial image corresponding to the underexposed or overexposed area of the facial image to be enhanced is filled with the corresponding delicate and natural-toned skin texture details, thereby improving the problem of missing portrait details in underexposed or overexposed scenes.
例如，当待增强图像存在因离焦、数字变焦或抖动造成的人像模糊、细节涂抹的情况下，采用上述优质的面部特征作为引导对该待增强面部图像进行增强处理，增强后的面部图像中与待增强图像的细节涂抹区域对应的区域会包含更多的纹理细节信息，增强后的面部图像的整体会包含更细腻、色调更自然的皮肤纹理，也即人像模糊问题也可以得到改善，从而可以改善因离焦、数字变焦或抖动造成的人像模糊、细节涂抹的问题。For example, when the image to be enhanced has blurred portraits or smeared details due to defocus, digital zoom or shaking, the above-mentioned high-quality facial features are used as a guide to enhance the facial image to be enhanced. The area in the enhanced facial image corresponding to the smeared detail area of the image to be enhanced will contain more texture detail information, and the enhanced facial image as a whole will contain a more delicate skin texture with a more natural tone, that is, the problem of blurred portraits and smeared details caused by defocus, digital zoom or shaking can also be improved.
S511，电子设备基于增强后的五官图像和增强后的面部图像融合得到增强肖像A。S511, the electronic device obtains an enhanced portrait A based on the fusion of the enhanced facial features image and the enhanced face image.
为便于描述，以下将第一图像的肖像中除了面部图像和五官图像之外的其他部分图像简称为剩余图像。For the convenience of description, the other partial images of the portrait of the first image except the facial image and the facial features image are referred to as the remaining images below.
在本申请实施例中，对增强后的五官图像、增强后的面部图像、以及剩余图像进行图像融合具体可以包括：将该增强后的五官图像、增强后的面部图像、以及剩余图像进行拼图处理，并对拼图的边缘进行平滑处理。In an embodiment of the present application, image fusion of the enhanced facial features image, the enhanced facial image, and the remaining image may specifically include: puzzle processing the enhanced facial features image, the enhanced facial image, and the remaining image, and smoothing the edges of the puzzle.
示例性的，电子设备基于图像融合方法对增强后的五官图像、增强后的面部图像、以及剩余图像进行图像融合以及色彩校正，得到上述增强肖像A。示例性的，图像融合方法可以有泊松图像融合、色彩迁移图像融合、拉普拉斯金字塔图像融合等方法，例如可以通过自适应实例标准化(adaptive instance normalization，AdaIN)方法实现风格迁移，对增强后的五官图像、增强后的面部图像、以及剩余图像进行图像融合，合成得到上述增强肖像A。Exemplarily, the electronic device performs image fusion and color correction on the enhanced facial features image, the enhanced facial image, and the remaining image based on an image fusion method to obtain the enhanced portrait A. Exemplarily, the image fusion method may include Poisson image fusion, color migration image fusion, Laplace pyramid image fusion, and the like. For example, style migration may be achieved through an adaptive instance normalization (AdaIN) method, and the enhanced facial features image, the enhanced facial image, and the remaining image are image fused to synthesize the enhanced portrait A.
在本申请实施例中，电子设备基于图像融合方法对增强后的五官图像、增强后的面部图像、以及剩余图像进行图像融合，既可以将增强后的五官图像和增强后的面部图像进行拼图以及对拼图边缘进行平滑处理，还可以基于该增强后的五官图像和该增强后的面部图像的色彩信息对该剩余图像进行色彩校正，色彩校正后的该剩余图像(例如衣服区域的图像)的清晰度、光照色彩与增强后的脸部图像一致。In an embodiment of the present application, the electronic device performs image fusion on the enhanced facial features image, the enhanced facial image, and the remaining image based on an image fusion method. It can not only puzzle the enhanced facial features image and the enhanced facial image and smooth the puzzle edges, but also perform color correction on the remaining image based on the color information of the enhanced facial features image and the enhanced facial image. The clarity and lighting color of the remaining image (for example, the image of the clothing area) after color correction are consistent with the enhanced facial image.
在另外一种可能的实现方式中，融合得到上述增强肖像A后，电子设备还可以对该增强肖像A中的第一区域进行色彩校正，该第一区域包括增强肖像A中色彩不均匀的区域、光照不协调的区域、欠曝区域、过曝区域中的一项或一项以上。In another possible implementation, after fusing the enhanced portrait A, the electronic device may further perform color correction on a first region in the enhanced portrait A, where the first region includes one or more of a region with uneven color, a region with uncoordinated lighting, an underexposed region, and an overexposed region in the enhanced portrait A.
例如，增强肖像A的五官图像基于第一优质五官图像增强得到，该第一优质五官图像中的每个五官区域图像可能来源于同一人物的多种不同图像，也就是说第一优质五官图像中的每个单独的五官区域图像之间的光照、色彩、曝光信息会有所差异，则增强后的各个五官图像之间也可能会存在明显的光照不均匀、色彩不均匀或过曝欠曝问题，基于此，在得到上述增强肖像A后，再对增强肖像A中的色彩不均匀的区域、光照不协调的区域、欠曝区域、过曝区域进行色彩校正，改善增强后的各个五官图像之间也可能会存在明显的光照不均匀、色彩不均匀、过曝、或欠曝的问题。For example, the facial feature image of the enhanced portrait A is obtained based on the enhancement of the first high-quality facial feature image. Each facial feature area image in the first high-quality facial feature image may come from multiple different images of the same person. That is to say, the lighting, color, and exposure information between each individual facial feature area image in the first high-quality facial feature image may be different. In this case, there may be obvious lighting unevenness, color unevenness, or overexposure or underexposure problems between the enhanced facial feature images. Based on this, after obtaining the above-mentioned enhanced portrait A, color correction is performed on the color uneven areas, lighting inharmonious areas, underexposed areas, and overexposed areas in the enhanced portrait A to improve the obvious lighting unevenness, color unevenness, overexposure, or underexposure problems between the enhanced facial feature images.
示例性的，电子设备可以基于增强肖像A的HSV颜色模型，确定增强肖像A中是否存在色彩或光照不均匀现象、过曝或欠曝现象、以及具体的色彩不均匀的区域、光照不协调的区域、欠曝或过曝区域。或者也可以基于深度学习的方法构建曝光检测模型，将第一图像输入该曝光检测模型，该曝光检测模型可以输出第一图像的曝光区域的语义分割mask，以此确定第一图像中是否存在过曝或欠曝现象、以及过曝区域或欠曝区域。For example, the electronic device can determine whether there is color or lighting unevenness, overexposure or underexposure, and specific color uneven areas, lighting uncoordinated areas, underexposure or overexposure areas in the enhanced portrait A based on the HSV color model of the enhanced portrait A. Alternatively, an exposure detection model can be constructed based on a deep learning method, and the first image can be input into the exposure detection model. The exposure detection model can output a semantic segmentation mask of the exposure area of the first image to determine whether there is overexposure or underexposure, and overexposure areas or underexposure areas in the first image.
示例性的，对该增强肖像A中的欠曝或过曝区域进行色彩校正(也可以理解为曝光矫正)可以包括：基于深度学习方法使用曝光矫正模型对该增强肖像A进行曝光矫正。例如，使用大量欠曝或过曝的图像作为训练集，基于曝光矫正模型中间输出的曝光区域的语义分割mask的准确性、以及与欠曝或过曝图像对应的曝光情况良好的图像作为约束集，设计该曝光矫正模型的损失函数，基于深度学习方法获得该曝光矫正模型。或者，对该增强肖像A中的欠曝或过曝区域进行曝光矫正还可以是基于泊松融合方法对增强肖像A进行曝光矫正，本文对此不做限定。或者，也可以基于其他本领域技术人员熟知的曝光矫正方法对增强肖像A进行曝光矫正，本文对此不做限定。Exemplarily, color correction (which can also be understood as exposure correction) of the underexposed or overexposed area in the enhanced portrait A may include: using an exposure correction model based on a deep learning method to perform exposure correction on the enhanced portrait A. For example, a large number of underexposed or overexposed images are used as a training set, and the accuracy of the semantic segmentation mask of the exposure area output by the exposure correction model and the image with good exposure corresponding to the underexposed or overexposed image are used as a constraint set to design the loss function of the exposure correction model, and obtain the exposure correction model based on a deep learning method. Alternatively, exposure correction of the underexposed or overexposed area in the enhanced portrait A may also be performed based on a Poisson fusion method to perform exposure correction on the enhanced portrait A, which is not limited herein. Alternatively, exposure correction of the enhanced portrait A may also be performed based on other exposure correction methods well known to those skilled in the art, which is not limited herein.
阶段4：电子设备将增强肖像A与背景图像进行背景融合增强，以获得图像处理后的第二图像。Stage 4: The electronic device performs background fusion enhancement on the enhanced portrait A and the background image to obtain a second image after image processing.
S512，电子设备对增强肖像A和第一图像的背景图像进行背景融合增强得到第二图像。S512: The electronic device performs background fusion enhancement on the enhanced portrait A and the background image of the first image to obtain a second image.
在本申请实施例中，对增强肖像A和第一图像的背景图像进行背景融合增强具体包括：对增强肖像A和第一图像的背景图像进行拼图处理，并对该增强肖像A和该背景图像的拼图进行边缘平滑处理，消除拼图边缘突兀感，以使得增强后得到的图像更自然。In an embodiment of the present application, background fusion enhancement of the enhanced portrait A and the background image of the first image specifically includes: performing puzzle processing on the enhanced portrait A and the background image of the first image, and performing edge smoothing processing on the puzzle of the enhanced portrait A and the background image to eliminate the abruptness of the puzzle edges, so as to make the enhanced image more natural.
在一些可能的实现方式中，在增强肖像A与第一图像的原始背景图像之间满足色彩不均匀条件或满足光照不协调条件的情况下，电子设备还可以利用第一增强肖像对第一图像的原始背景图像进行色彩校正，以使得增强后得到的图像更自然。In some possible implementations, when the enhanced portrait A and the original background image of the first image satisfy a color unevenness condition or a lighting inharmoniousness condition, the electronic device may also use the first enhanced portrait to perform color correction on the original background image of the first image to make the enhanced image more natural.
例如，电子设备基于人物处理子系统30中的图像特征提取，提取增强肖像A和第一图像的背景图像的颜色特征，基于色彩迁移技术、自适应实例标准化(AdaIN)、风格迁移网络、或光照校准网络技术，利用该增强肖像A的色彩信息对该背景图像的色彩进行矫正，进一步减小增强肖像A与背景图像之间的色彩差异(或称光照色彩差异)，使得合成后的第二图像更自然。For example, the electronic device extracts color features of the enhanced portrait A and the background image of the first image based on image feature extraction in the character processing subsystem 30, and corrects the color of the background image using the color information of the enhanced portrait A based on color migration technology, adaptive instance normalization (AdaIN), style migration network, or illumination calibration network technology, thereby further reducing the color difference (or illumination color difference) between the enhanced portrait A and the background image, making the synthesized second image more natural.
例如，第一图像为暗光场景下模糊的、细节缺失的图像，增强肖像B为第一图像进行五官增强、面部增强处理过的、光照信息较佳的图像，而背景图像则未进行增强处理、光照信息较差(例如模糊)的图像，则电子设备可以基于该增强肖像B的像素点的色彩信息调整该背景图像的像素点的像素值，协调图像整体色差。For example, the first image is a blurred image with missing details in a dark scene, the enhanced portrait B is an image of the first image with facial features and face enhancement processed and with better lighting information, while the background image is not enhanced and has poor lighting information (for example, blurred). The electronic device can adjust the pixel values of the pixels of the background image based on the color information of the pixels of the enhanced portrait B to coordinate the overall color difference of the image.
在本申请实施例中，电子设备获得上述第二图像后，可以删除已存储的第一图像的相关数据(包含第一图像的原始图像、人像抠图语义、人脸解析语义、人脸本征信息、头部姿态、五官评分结果、以及面部评分结果)，并将所述第一图像的索引与所述第二图像建立关联关系，将第二图像作为历史图像存储图像数据库50中。以及，将第二图像设置为已进行过图像增强的图像(具体可以通过为第二图像添加预设标识的方式实现)，用以表示该第二图像为已进行图像增强过的图像，以使得该第二图像不会再作为其他待增强图像的优质图像，也即不会利用第二图像对其他待增强图像进行增强处理。In the embodiment of the present application, after the electronic device obtains the second image, it can delete the stored relevant data of the first image (including the original image of the first image, the semantics of portrait cutout, the semantics of face analysis, the intrinsic information of the face, the head posture, the score result of the facial features, and the facial score result), and establish an association relationship between the index of the first image and the second image, and store the second image as a historical image in the image database 50. In addition, the second image is set as an image that has been image enhanced (specifically, it can be achieved by adding a preset mark to the second image) to indicate that the second image is an image that has been image enhanced, so that the second image will no longer be used as a high-quality image of other images to be enhanced, that is, the second image will not be used to enhance other images to be enhanced.
在另外一些可能的实现方式中，电子设备还可以对第一图像的脖子区域对应的图像和/或头发区域对应的图像做单独增强处理。示例性的，电子设备可以基于人脸解析获取第一图像中的脖子和头发部分对应的图像。其中，对第一图像的脖子区域对应的图像做单独增强处理可以包括但不限于：对脖子区域对应的图像进行清晰度和纹理细腻度增强。对第一图像的头发区域对应的图像做单独增强处理可以。包括但不限于：对头发的光泽度进行增强处理，对头发区域的对比度进行增强处理。In some other possible implementations, the electronic device may also perform separate enhancement processing on the image corresponding to the neck area and/or the image corresponding to the hair area of the first image. Exemplarily, the electronic device may obtain images corresponding to the neck and hair parts in the first image based on face analysis. The separate enhancement processing on the image corresponding to the neck area of the first image may include, but is not limited to: enhancing the clarity and texture fineness of the image corresponding to the neck area. The separate enhancement processing on the image corresponding to the hair area of the first image may include, but is not limited to: enhancing the glossiness of the hair and enhancing the contrast of the hair area.
在本申请实施例中，对图像进行相应的增强是指对图像进行相应的处理使得图像质量变得更好，例如由差变好。例如对图像的纹理进行增强是指：将纹理不清晰的图像增强为纹理更为细腻的图像；对图像的曝光情况进行增强是指：将图像中的过曝或欠曝区域，增强为曝光正常或较正常的图像，也即增强是指使得图像质量变得更好。In the embodiments of the present application, enhancing the image accordingly means processing the image accordingly so that the image quality becomes better, for example, from bad to good. For example, enhancing the texture of an image means: enhancing an image with unclear texture to an image with finer texture; enhancing the exposure of an image means: enhancing an overexposed or underexposed area in an image to an image with normal or relatively normal exposure, that is, enhancing means making the image quality better.
以下结合图6所示的方法流程图详细说明上述图2所示的图像处理方法中利用优质肖像或利用优质肖像和第二优质五官图像对第一图像进行增强的一种具体实现方式。如图6所示，该图像处理方法可以包括以下几个阶段和对应的方法步骤。A specific implementation of enhancing the first image by using the high-quality portrait or the high-quality portrait and the second high-quality facial features image in the image processing method shown in FIG2 is described in detail below in conjunction with the method flow chart shown in FIG6. As shown in FIG6, the image processing method may include the following stages and corresponding method steps.
阶段1：电子设备获取到第一图像后，确定第一图像是否包含人脸，在确定包含人脸的情况下，进入阶段2。Stage 1: After the electronic device acquires the first image, it determines whether the first image contains a face. If it is determined that the first image contains a face, it enters stage 2.
S601，电子设备获取第一图像。S601: The electronic device acquires a first image.
关于如何获取第一图像可以参照本文其他实施例的相关说明，例如上述步骤S501中的相关说明，在此不再详述。Regarding how to obtain the first image, reference may be made to the relevant descriptions of other embodiments of this document, such as the relevant descriptions in the above step S501, which will not be described in detail here.
S602，电子设备基于人脸检测确定第一图像是否存在人脸。S602: The electronic device determines whether there is a face in the first image based on face detection.
关于如何确定第一图像是否存在人脸可以参照本文其他实施例的相关说明，例如上述步骤S502，在此不再详述。Regarding how to determine whether there is a face in the first image, reference may be made to the relevant descriptions of other embodiments of this document, such as the above-mentioned step S502, which will not be described in detail here.
在确定第一图像存在人脸的情况下，电子设备执行步骤S603、S605、S606、S607；在确定第一图像不存在人脸的情况下，电子设备不解析第一图像也不对第一图像进行增强处理(在图6中以步骤S614示出)，也即不执行步骤S603至S612。When it is determined that there is a face in the first image, the electronic device executes steps S603, S605, S606, and S607; when it is determined that there is no face in the first image, the electronic device does not parse the first image or enhance the first image (shown as step S614 in Figure 6), that is, does not execute steps S603 to S612.
阶段2：电子设备获取第一图像的相关数据，具体包括：人脸聚类标签、人像抠图数据(包含待增强肖像和背景图像)、人脸解析数据、画质评分结果、五官评分结果、以及面部评分结果。Stage 2: The electronic device obtains relevant data of the first image, including: face clustering labels, portrait cutout data (including the portrait to be enhanced and the background image), face analysis data, image quality scoring results, facial features scoring results, and facial scoring results.
S603，电子设备获取第一图像对应的待增强肖像和背景图像。S603: The electronic device obtains a portrait image to be enhanced and a background image corresponding to the first image.
示例性的，电子设备通过人物处理子系统30中的人像处理模块的人像抠图功能获取第一图像对应的待增强肖像和背景图像。Exemplarily, the electronic device obtains the portrait to be enhanced and the background image corresponding to the first image through the portrait cutout function of the portrait processing module in the character processing subsystem 30 .
在本申请中，电子设备获取到第一图像的待增强肖像和背景图像后，可以将其存储在本地存储空间中，也可以存储到电子设备的云服务器端，本文对此不做限定。In the present application, after the electronic device obtains the portrait and background image to be enhanced of the first image, it can store them in a local storage space or in a cloud server of the electronic device, which is not limited in this article.
S604，电子设备获取第一图像的画质评分结果、五官评分结果以及面部评分结果。S604, the electronic device obtains the image quality scoring result, the facial features scoring result and the face scoring result of the first image.
在本申请实施例中，第一图像的画质评分的评分标准与评分区域对应的图像的清晰度相关，评分区域对应的图像的清晰度越好则画质评分结果越优，该评分区域可以是第一图像中整张图像，也可以是第一图像中的人像区域(也即肖像区域)。In an embodiment of the present application, the scoring criteria for the image quality score of the first image is related to the clarity of the image corresponding to the scoring area. The better the clarity of the image corresponding to the scoring area, the better the image quality score result. The scoring area can be the entire image in the first image, or it can be the portrait area (i.e., portrait area) in the first image.
示例性的，电子设备在拍摄照片时会自动标记当前拍照模式，例如自动模式、人像模式、夜景模式、专业模式等。对于不同的模式，在进行画质评分时，评分区域有所不同。示例性的，在进行画质评分时会读取当前照片的拍照模式，若为人像拍照模式，那么画质评分区域只对人像区域进行评价。若为自动拍照模式，那么评分区域为整张图像。可理解的，由于人像模式下人物图像中的背景图像默认均会有一定的模糊程度，则在计算该人物图像的模糊度时，不考虑该人物图像的背景图像的模糊度，得到的画质评分结果更具代表性。Exemplarily, the electronic device will automatically mark the current camera mode when taking a photo, such as automatic mode, portrait mode, night mode, professional mode, etc. For different modes, the scoring area is different when scoring the image quality. Exemplarily, the camera mode of the current photo will be read when scoring the image quality. If it is a portrait camera mode, the image quality scoring area will only evaluate the portrait area. If it is an automatic camera mode, the scoring area is the entire image. It is understandable that since the background image in the character image in portrait mode will have a certain degree of blur by default, the blur of the background image of the character image is not considered when calculating the blur of the character image, and the resulting image quality scoring result is more representative.
在本申请实施例中，第一图像的画质评分结果可以为与N种清晰度计算方法对应的N项清晰度值，N大于或等于2，该N项清晰度值中的每一个清晰度值均可以用于代表第一图像的清晰度。或者，第一图像的画质评分结果也可以为N项清晰度值对应的百分制或等级制结果，本文对此不做限定。关于清晰度值的计算方法参考上文相关描述，在此不再赘述。In the embodiment of the present application, the image quality scoring result of the first image may be N clarity values corresponding to N clarity calculation methods, where N is greater than or equal to 2, and each of the N clarity values may be used to represent the clarity of the first image. Alternatively, the image quality scoring result of the first image may also be a percentage or graded result corresponding to the N clarity values, which is not limited herein. For the calculation method of the clarity value, refer to the relevant description above and will not be repeated here.
上述N种清晰度值包括但不限于Brenner梯度函数、Laplacian梯度函数、SMD(灰度方差)函数、无参考图像评价指标(NIQE)、Brisque。其中，一般情况下，Brenner梯度函数、Laplacian梯度函数、以及SMD函数计算得到的值越大清晰度越好；NIQE的值越小，表示图像清晰度越好。可理解的，若仅采用一种清晰度计算方法评估图像的清晰度，评估结果可能会不太客观，基于此，可以综合清晰度的各种不同统计方法的计算结果评估图像的清晰度，进一步提高清晰度评估结果(例如画质评分结果)的客观性。上述有关清晰度计算的函数仅为示例，还可以有其他可以达到相同效果的算法或模型，其为本领域技术人员熟知的技术，在此不再叙述。The above-mentioned N kinds of clarity values include but are not limited to Brenner gradient function, Laplacian gradient function, SMD (grayscale variance) function, non-reference image evaluation index (NIQE), and Brisque. Among them, in general, the larger the value calculated by the Brenner gradient function, Laplacian gradient function, and SMD function, the better the clarity; the smaller the value of NIQE, the better the image clarity. It is understandable that if only one clarity calculation method is used to evaluate the clarity of an image, the evaluation result may not be very objective. Based on this, the calculation results of various different statistical methods of clarity can be combined to evaluate the clarity of the image, and the objectivity of the clarity evaluation results (such as image quality scoring results) can be further improved. The above-mentioned functions for clarity calculation are only examples, and there may be other algorithms or models that can achieve the same effect. They are technologies well known to those skilled in the art and will not be described here.
示例性的，电子设备在获取到第一图像的人脸解析数据后对第一图像的五官图像进行评分，以及对第一图像的面部图像进行评分。关于五官评分结果、面部评分结果、以及电子设备具体如何获取第一图像的五官评分结果和面部评分结果的说明，可以参照本文其他实施例的相关说明，在此不再详述，例如上述步骤S504。Exemplarily, after acquiring the face analysis data of the first image, the electronic device scores the facial features image of the first image and scores the facial image of the first image. For the description of the facial features scoring result, the facial scoring result, and how the electronic device specifically obtains the facial features scoring result and the facial scoring result of the first image, please refer to the relevant description of other embodiments of this document, which will not be described in detail here, such as the above step S504.
S605，电子设备获取第一图像的人脸聚类标签。S605: The electronic device obtains a face clustering label of the first image.
关于具体如何获取第一图像的人脸聚类标签可以参照本文其他实施例的相关说明(例如步骤S505)，在此不再详述。Regarding how to obtain the face clustering label of the first image, please refer to the relevant description of other embodiments of this document (such as step S505), which will not be described in detail here.
在本申请中，电子设备获取到第一图像的人脸聚类标签后，可以将其存储在本地存储空间中，也可以存储到云端，本文对此不做限定。In the present application, after the electronic device obtains the face clustering label of the first image, it can store it in a local storage space or in the cloud, which is not limited in this article.
S606，电子设备确定第一图像是否满足增强条件。S606: The electronic device determines whether the first image meets an enhancement condition.
关于具体如何确定第一图像是否满足增强条件可以参照本文其他实施例的相关说明(例如步骤S506)，在此不再详述。Regarding how to determine whether the first image satisfies the enhancement condition, reference may be made to the relevant descriptions of other embodiments of this document (eg, step S506 ), which will not be described in detail here.
在确定第一图像满足增强条件的情况下，电子设备执行步骤S607；在确定第一图像不满足增强条件的情况下，电子设备不对第一图像进行增强处理(在图6中以步骤S613示出)，也即不执行步骤S607至S612。If it is determined that the first image meets the enhancement condition, the electronic device executes step S607; if it is determined that the first image does not meet the enhancement condition, the electronic device does not enhance the first image (shown as step S613 in Figure 6), that is, does not execute steps S607 to S612.
阶段3：检索优质肖像并利用优质肖像对待增强肖像进行增强处理，得到增强肖像A。Stage 3: Retrieve the high-quality portrait and use it to enhance the portrait to be enhanced, thereby obtaining the enhanced portrait A.
S607，电子设备获取与第一图像的待增强肖像匹配的优质肖像。S607: The electronic device obtains a high-quality portrait that matches the portrait to be enhanced in the first image.
电子设备中或电子设备的云服务器中(例如图像数据库50中)存储有历史人物图像的人脸聚类标签、人像抠图数据(包含待增强肖像和背景图像)、人脸解析数据、画质评分结果、五官评分结果、面部评分结果、人脸本征信息、以及头部姿态。其中，该历史人物图像是指电子设备在获取到第一图像之前存储的。The electronic device or the cloud server of the electronic device (e.g., the image database 50) stores face clustering labels, portrait cutout data (including the portrait to be enhanced and the background image), face analysis data, image quality rating results, facial features rating results, facial rating results, face intrinsic information, and head posture of the historical figure image. The historical figure image refers to the image stored by the electronic device before the first image is acquired.
以下提供三种选取优质肖像的方式：Here are three ways to select a good quality portrait:
方式1：电子设备可以基于历史人物图像的画质评分结果、五官评分结果、以及面部评分结果选取上述优质肖像。Method 1: The electronic device may select the above-mentioned high-quality portrait based on the image quality scoring results, the facial features scoring results, and the face scoring results of the historical figure images.
示例性的，电子设备可以先基于第一图像的人脸聚类标签和第一图像的画质评分结果确定肖像图像集合A，再确定该肖像图像集合A中评分最优的肖像图像作为上述优质肖像，该肖像集合A中的任意一个肖像图像与待增强肖像来源于相同的目标人物，该肖像图像集合A中的任意一个肖像图像的画质评分结果满足预设画质评分阈值。例如，该肖像图像集合A中的任意一个肖像图像对应的画质评分结果包含N种清晰度算法对应的N个清晰度值，则该N个清晰度值均满足对应的预设清晰度阈值。上述确定该肖像图像集合A中评分最优的肖像图像作为上述优质肖像包括：基于肖像评分函数和相应的加权系数将肖像图像集合A中的每一个肖像图像对应的面部评分结果和五官评分结果综合为一个评分值作为排序依据，基于综合的评分值选取该肖像图像集合A中评分最优的肖像图像作为该优质图像。Exemplarily, the electronic device may first determine the portrait image set A based on the face clustering label of the first image and the image quality scoring result of the first image, and then determine the portrait image with the best score in the portrait image set A as the above-mentioned high-quality portrait, wherein any portrait image in the portrait image set A and the portrait to be enhanced are from the same target person, and the image quality scoring result of any portrait image in the portrait image set A meets the preset image quality scoring threshold. For example, the image quality scoring result corresponding to any portrait image in the portrait image set A includes N clarity values corresponding to N clarity algorithms, and the N clarity values all meet the corresponding preset clarity threshold. The above-mentioned determination of the portrait image with the best score in the portrait image set A as the above-mentioned high-quality portrait includes: based on the portrait scoring function and the corresponding weighting coefficient, the facial scoring result and the facial features scoring result corresponding to each portrait image in the portrait image set A are integrated into a scoring value as a ranking basis, and the portrait image with the best score in the portrait image set A is selected as the high-quality image based on the integrated scoring value.
方式2：电子设备基于历史人物图像的画质评分结果、五官评分结果、面部评分结果、以及头部姿态旋转角选取上述优质肖像。Method 2: The electronic device selects the above-mentioned high-quality portrait based on the image quality scoring results, facial features scoring results, face scoring results, and head posture rotation angle of the historical figure image.
示例性的，电子设备可以先基于第一图像的人脸聚类标签和第一图像的画质评分结果确定肖像图像集合A，再确定该肖像图像集合A中评分较优且肖像图像的头部姿态旋转角与第一图像的肖像图像的头部姿态旋转角差异较小的肖像图像作为上述优质肖像。例如，该肖像集合A中的任意一个肖像图像对应的画质评分结果包含N种清晰度算法对应的N个清晰度值，则该N个清晰度值均满足对应的预设清晰度阈值。上述确定该肖像图像集合A中评分较优且肖像图像的头部姿态旋转角与第一图像的肖像图像的头部姿态旋转角差异较小的肖像图像作为上述优质肖像包括：基于第一图像的肖像图像的头部姿态旋转角确定肖像图像集合B，再确定该肖像图像集合B中评分最优的肖像图像作为上述优质肖像，该肖像图像集合B中的任意一个肖像图像来源于该肖像图像集合A，且该肖像图像集合B中的任意一个肖像图像的头部姿态旋转角与第一图像的肖像图像的头部姿态旋转角的差值小于预设欧拉角阈值，上述确定该肖像图像集合B中评分最优的肖像图像作为上述优质肖像包括：基于肖像评分函数和相应的加权系数将肖像图像集合B中的每一个肖像图像对应的面部评分结果和五官评分结果综合为一个评分值作为排序依据，基于综合的评分值选取该肖像图像集合B中评分最优的肖像图像作为该优质图像。Exemplarily, the electronic device may first determine the portrait image set A based on the face clustering label of the first image and the image quality scoring result of the first image, and then determine the portrait image with a better score in the portrait image set A and a smaller difference in the head posture rotation angle between the portrait image and the head posture rotation angle of the portrait image of the first image as the above-mentioned high-quality portrait. For example, if the image quality scoring result corresponding to any portrait image in the portrait set A includes N clarity values corresponding to N clarity algorithms, then the N clarity values all meet the corresponding preset clarity thresholds. The above-mentioned determination of the portrait image with a higher score in the portrait image set A and the smaller difference between the head posture rotation angle of the portrait image and the head posture rotation angle of the portrait image of the first image as the above-mentioned high-quality portrait includes: determining the portrait image set B based on the head posture rotation angle of the portrait image of the first image, and then determining the portrait image with the best score in the portrait image set B as the above-mentioned high-quality portrait, any portrait image in the portrait image set B originates from the portrait image set A, and the difference between the head posture rotation angle of any portrait image in the portrait image set B and the head posture rotation angle of the portrait image of the first image is less than a preset Euler angle threshold, and the above-mentioned determination of the portrait image with the best score in the portrait image set B as the above-mentioned high-quality portrait includes: based on the portrait scoring function and the corresponding weighting coefficient, the facial scoring results and the facial features scoring results corresponding to each portrait image in the portrait image set B are integrated into a scoring value as a sorting basis, and the portrait image with the best score in the portrait image set B is selected as the high-quality image based on the integrated scoring value.
方式3：电子设备先基于上述方式1选取优质肖像(例如记为优质肖像A)，再确定该优质肖像A与第一图像的头部姿态旋转角的差值是否小于预设欧拉角阈值，若是，则电子设备确定基于该优质肖像A对待增强肖像进行图像增强；若该优质肖像A与第一图像的头部姿态旋转角的差值大于或等于预设欧拉角阈值，则电子设备基于方式2重新选取优质肖像(例如记为优质肖像B)，基于该优质肖像B对待增强肖像进行图像增强。Method 3: The electronic device first selects a high-quality portrait (for example, recorded as high-quality portrait A) based on the above-mentioned method 1, and then determines whether the difference between the high-quality portrait A and the head posture rotation angle of the first image is less than the preset Euler angle threshold. If so, the electronic device determines to perform image enhancement on the portrait to be enhanced based on the high-quality portrait A; if the difference between the high-quality portrait A and the head posture rotation angle of the first image is greater than or equal to the preset Euler angle threshold, the electronic device re-selects a high-quality portrait (for example, recorded as high-quality portrait B) based on method 2, and performs image enhancement on the portrait to be enhanced based on the high-quality portrait B.
在本申请实施例中，上述肖像图像集合A、肖像图像集合B中的任意一个肖像图像对应的画质评分结果具体可以是该肖像图像对应的人物图像的整体图像的画质评分结果，也可以是该肖像图像对应的人物图像的肖像图像区域的画质评分结果。In an embodiment of the present application, the image quality scoring result corresponding to any portrait image in the above-mentioned portrait image set A or portrait image set B can specifically be the image quality scoring result of the overall image of the character image corresponding to the portrait image, or it can be the image quality scoring result of the portrait image area of the character image corresponding to the portrait image.
基于第一图像的人脸聚类标签、第一图像的画质评分结果、五官评分结果、面部评分结果、以及图像数据库50，获取与上述待增强肖像匹配的优质肖像。例如，可以将与第一图像中对应的人脸聚类标签一致的历史图像的画质评分结果、面部评分结果、以及五官评分结果的加权和作为该历史图像的综合肖像评分，基于综合肖像评分获取评分最高的历史图像的肖像作为上述优质肖像。具体的加权数可以根据实际情况调整，本文对此不做限定。Based on the face clustering label of the first image, the image quality scoring result, the facial features scoring result, the face scoring result of the first image, and the image database 50, a high-quality portrait matching the above-mentioned portrait to be enhanced is obtained. For example, the weighted sum of the image quality scoring result, the facial scoring result, and the facial features scoring result of the historical image that is consistent with the corresponding face clustering label in the first image can be used as the comprehensive portrait score of the historical image, and the portrait of the historical image with the highest score is obtained based on the comprehensive portrait score as the above-mentioned high-quality portrait. The specific weighting number can be adjusted according to the actual situation, and this document does not limit this.
S608，电子设备利用优质肖像对待增强肖像进行增强处理，获得增强肖像A。S608 , the electronic device uses the high-quality portrait to enhance the portrait to be enhanced, to obtain an enhanced portrait A.
例如，可以采用深度学习方法在隐编码层利用优质肖像对待增强肖像进行增强处理。具体的，将优质肖像和该待增强肖像输入肖像增强网络模型，该肖像增强网络模型对优质肖像和该待增强肖像进行特征提取，基于提取到的特征信息进行特征融合计算，输出增强肖像A。其中，该肖像增强网络模型基于样本数据训练得到。For example, a deep learning method can be used to enhance the portrait to be enhanced using the high-quality portrait in the hidden coding layer. Specifically, the high-quality portrait and the portrait to be enhanced are input into a portrait enhancement network model, and the portrait enhancement network model extracts features from the high-quality portrait and the portrait to be enhanced, performs feature fusion calculation based on the extracted feature information, and outputs an enhanced portrait A. The portrait enhancement network model is trained based on sample data.
在本申请实施例中，肖像增强模型可以采用有监督学习方法或无监督学习方法训练得到，本文对此不做限定。In the embodiment of the present application, the portrait enhancement model can be trained using a supervised learning method or an unsupervised learning method, which is not limited herein.
示例性的，采用有监督学习方法和至少两组实验数据训练得到肖像增强模型，具体的，实验数据包括第一样本图像、第二样本图像、以及第三样本图像，该第一样本图像和该第二样本图像作为该肖像增强模型的输入，该第三样本图像作为第一输出图像的约束图像，该第一输出图像为该第一肖像增强模型中输出的与该第一样本图像对应的增强处理后的图像，该第一样本图像和该第三样本图像包含相同肖像的两张不同图像质量的肖像，该第三样本图像的图像质量优于该第一样本图像的图像质量，该第二样本图像包含第一参考肖像、且该第二样本图像的图像质量优于该第一样本图像，该第一样本图像包含的肖像图像与该第一参考肖像对应同一人物；或者，Exemplarily, a supervised learning method and at least two groups of experimental data are used to train a portrait enhancement model. Specifically, the experimental data include a first sample image, a second sample image, and a third sample image. The first sample image and the second sample image are used as inputs of the portrait enhancement model, and the third sample image is used as a constraint image of a first output image. The first output image is an enhanced image corresponding to the first sample image and output in the first portrait enhancement model. The first sample image and the third sample image include two portraits of the same portrait with different image qualities. The image quality of the third sample image is better than that of the first sample image. The second sample image includes a first reference portrait, and the image quality of the second sample image is better than that of the first sample image. The portrait image included in the first sample image corresponds to the same person as the first reference portrait. Or,
示例性的，采用无监督学习方法和至少两组实验数据训练得到肖像增强模型，具体的，该实验数据包括第四样本图像和第五样本图像，该肖像增强模型包括生成模型和对抗模型，该生成模型用于以该第五样本图像作为指导图像生成与该第四样本图像对应的增强处理后的第二输出图像，该对抗模型为预训练好的用于评判该第二输出图像是否符合增强效果的评判网络，该第五样本图像包含第二参考肖像、且该第五样本图像的图像质量优于该第四样本图像，该第四样本图像包含的肖像图像与该第二参考肖像对应同一人物。Exemplarily, an unsupervised learning method and at least two sets of experimental data are used to train a portrait enhancement model. Specifically, the experimental data include a fourth sample image and a fifth sample image. The portrait enhancement model includes a generative model and an adversarial model. The generative model is used to generate a second output image after enhancement corresponding to the fourth sample image using the fifth sample image as a guidance image. The adversarial model is a pre-trained judgment network for judging whether the second output image meets the enhancement effect. The fifth sample image includes a second reference portrait, and the image quality of the fifth sample image is better than that of the fourth sample image. The portrait image included in the fourth sample image corresponds to the same person as the second reference portrait.
在一种可能的实现方式中，电子设备还可以对增强肖像A中的第二区域进行色彩校正，该第二区域包括增强肖像A中色彩不均匀的区域、光照不协调的区域、欠曝区域、过曝区域中的一项或一项以上。具体可以参照图5中S511中关于对增强肖像A中的第一区域进行色彩校正的相关说明。In a possible implementation, the electronic device may further perform color correction on a second area in the enhanced portrait A, where the second area includes one or more of an area with uneven color, an area with uncoordinated lighting, an underexposed area, and an overexposed area in the enhanced portrait A. For details, reference may be made to the description of performing color correction on the first area in the enhanced portrait A in S511 in FIG.
阶段4：对增强肖像A与背景图像进行背景融合增强，以获得图像处理后的第二图像。Stage 4: Background fusion enhancement is performed on the enhanced portrait A and the background image to obtain a second image after image processing.
S609，电子设备对增强肖像A和第一图像的背景图像进行背景融合增强得到第二图像。S609: The electronic device performs background fusion enhancement on the enhanced portrait A and the background image of the first image to obtain a second image.
具体可以参照本文其他实施例的相关说明(例如步骤S512)，在此不再详述。For details, please refer to the relevant descriptions of other embodiments in this document (such as step S512), which will not be described in detail here.
在另外一些可能的实现方式中，电子设备还可以对第一图像的脖子区域和/或头发区域对应的图像进行单独增强，具体可以参照上述实施例1中的相关说明，在此不再详述。In some other possible implementations, the electronic device may also separately enhance the image corresponding to the neck area and/or the hair area of the first image. For details, please refer to the relevant description in the above embodiment 1, which will not be described in detail here.
在另外一种可能的实现方式中，若上述优质肖像的头部姿态与待增强肖像的头部姿态相差较大，例如头部姿态差值大于或等于预设欧拉角阈值(例如预设欧拉角阈值可以为10或20，本文对此不做限定)，则电子设备可以基于三维人脸重建技术或NerfGAN技术对优质肖像进行头部姿态矫正，基于矫正后肖像图像对上述待增强肖像进行增强处理。In another possible implementation, if the head posture of the above-mentioned high-quality portrait is significantly different from the head posture of the portrait to be enhanced, for example, the head posture difference is greater than or equal to a preset Euler angle threshold (for example, the preset Euler angle threshold may be 10 or 20, which is not limited in this document), the electronic device may perform head posture correction on the high-quality portrait based on three-dimensional face reconstruction technology or NerfGAN technology, and enhance the above-mentioned portrait to be enhanced based on the corrected portrait image.
示例性的，电子设备可以对该优质肖像进行3D人脸重建，基于待增强肖像的头部姿态对3D人脸模型进行头部姿态矫正，利用矫正后的3D人脸模型对应的2D图像对待增强肖像进行增强处理，以下结合如图6中S610至S612所示的步骤加以说明。具体的，在步骤S608电子设备获取到优质肖像后，执行如下步骤：Exemplarily, the electronic device can perform 3D face reconstruction on the high-quality portrait, perform head posture correction on the 3D face model based on the head posture of the portrait to be enhanced, and enhance the portrait to be enhanced using the 2D image corresponding to the corrected 3D face model, as described below in conjunction with the steps shown in S610 to S612 in FIG. 6. Specifically, after the electronic device obtains the high-quality portrait in step S608, it executes the following steps:
S610，电子设备确定优质肖像的头部姿态与第一图像的头部姿态的差值是否大于或等于预设欧拉角阈值。S610: The electronic device determines whether a difference between a head posture of the high-quality portrait and a head posture of the first image is greater than or equal to a preset Euler angle threshold.
例如，电子设备可以通过调用人物处理子系统30中的人脸属性估计模型中的头部姿态估计对优质肖像和第一图像的头部姿态进行估计。若优质肖像的头部姿态与第一图像的头部姿态的差值小于预设欧拉角阈值，则电子设备通过执行步骤S609至S610以获得增强处理后的与第一图像对应的第二图像。若优质肖像的头部姿态与第一图像的头部姿态的差值大于或等于预设欧拉角阈值，则电子设备执行步骤S611。For example, the electronic device can estimate the head posture of the high-quality portrait and the first image by calling the head posture estimation in the face attribute estimation model in the character processing subsystem 30. If the difference between the head posture of the high-quality portrait and the head posture of the first image is less than the preset Euler angle threshold, the electronic device performs steps S609 to S610 to obtain a second image corresponding to the first image after enhancement processing. If the difference between the head posture of the high-quality portrait and the head posture of the first image is greater than or equal to the preset Euler angle threshold, the electronic device performs step S611.
S611，电子设备对优质肖像进行3D人脸重建，获得3D人脸模型。S611, the electronic device performs 3D face reconstruction on the high-quality portrait to obtain a 3D face model.
例如，电子设备基于优质肖像、优质肖像的人脸本征信息、优质肖像的纹理特征、边缘特征以及空间几何特征、对优质肖像进行3D人脸重建。例如优质肖像的人脸本征信息可以通过人物特征检索子系统20中的人脸属性特征参数检索得到或通过调用人物处理子系统30中的人脸属性估计模型中的人脸本征估计对优质肖像进行人脸本征估计得到。优质肖像的纹理特征、边缘特征以及空间几何特征可以通过人物处理子系统30中的特征提取模块提取得到。For example, the electronic device performs 3D face reconstruction on the high-quality portrait based on the high-quality portrait, the facial intrinsic information of the high-quality portrait, the texture features, the edge features and the spatial geometric features of the high-quality portrait. For example, the facial intrinsic information of the high-quality portrait can be retrieved by the facial attribute feature parameter in the character feature retrieval subsystem 20 or by calling the facial intrinsic estimation in the facial attribute estimation model in the character processing subsystem 30 to estimate the facial intrinsics of the high-quality portrait. The texture features, edge features and spatial geometric features of the high-quality portrait can be extracted by the feature extraction module in the character processing subsystem 30.
S612，电子设备基于待增强图像的头部姿态对3D人脸模型进行头部姿态矫正，并将矫正后的3D人脸模型渲染为2D图像，获得矫正后的优质肖像。S612, the electronic device performs head posture correction on the 3D face model based on the head posture of the image to be enhanced, and renders the corrected 3D face model into a 2D image to obtain a corrected high-quality portrait.
关于具体如何进行3D人脸重建、如何旋转3D人脸模型的头部姿态欧拉角进行头部姿态矫正、以及如何将矫正后的3D人脸模型渲染为2D图像，为本领域技术人员熟知的技术，在此不再详述。How to perform 3D face reconstruction, how to rotate the head posture Euler angles of the 3D face model to correct the head posture, and how to render the corrected 3D face model into a 2D image are technologies well known to those skilled in the art and will not be described in detail here.
电子设备在执行完S612获取到矫正后的优质肖像后，基于该矫正后的优质肖像执行S608至S609以获得第二图像。其中，在步骤S608中，电子设备利用该矫正后的优质肖像进行增强处理，获得上述增强肖像A。After executing S612 to obtain the corrected high-quality portrait, the electronic device executes S608 to S609 based on the corrected high-quality portrait to obtain a second image. In step S608, the electronic device performs enhancement processing using the corrected high-quality portrait to obtain the enhanced portrait A.
在一种可能的实现方式中，电子设备还可以检索与第一图像匹配的第二优质五官图像，在上述步骤S608之后，电子设备可以利用该第二优质五官图像对增强肖像(例如增强肖像A)进行五官增强处理。In one possible implementation, the electronic device may also retrieve a second high-quality facial feature image that matches the first image. After the above step S608, the electronic device may use the second high-quality facial feature image to perform facial feature enhancement processing on the enhanced portrait (eg, enhanced portrait A).
如图7所示，电子设备在步骤S608之后，执行如下步骤：As shown in FIG. 7 , after step S608, the electronic device performs the following steps:
S615，电子设备获取第二优质五官图像。S615, the electronic device obtains a second high-quality facial features image.
在本申请实施例中，第二优质五官图像可以是比第一图像中对应的五官图像的图像质量更优的五官图像，也可以是比增强肖像A中对应的五官图像的图像质量更优的五官图像。In an embodiment of the present application, the second high-quality facial feature image can be a facial feature image with better image quality than the corresponding facial feature image in the first image, or it can be a facial feature image with better image quality than the corresponding facial feature image in the enhanced portrait A.
在本申请实施例中，图7所示的步骤S615在步骤S608之后执行仅为示例，该步骤S615与S608可以同时执行也可以先后执行，且本文对其先后执行的顺序也不做限定。例如，步骤S615也可以在‘步骤S602之后以及步骤S608之前’的任意一个步骤执行，本文对此不做限定。In the embodiment of the present application, the execution of step S615 after step S608 shown in FIG. 7 is only an example, and step S615 and step S608 can be executed simultaneously or successively, and the order of their execution is not limited in this document. For example, step S615 can also be executed in any step of 'after step S602 and before step S608', and this document does not limit this.
S616，电子设备利用第二优质五官图像对增强肖像A进行五官单独增强处理，得到五官单独增强后的增强肖像A。S616, the electronic device uses the second high-quality facial feature image to perform facial feature enhancement processing on the enhanced portrait A to obtain the enhanced portrait A after the facial features are enhanced.
示例性的，上述第二优质五官图像包含第二鼻子图像、第二嘴巴图像、第二眉毛图像、具体的，利用第二优质五官图像对增强肖像A进行五官单独增强处理可以包括但不限于：基于该第二嘴巴图像对增强肖像A中的嘴唇区域的图像进行色彩和纹理增强处理；基于该第二眉毛图像对增强肖像A中的眉毛区域的图像进行色彩对比度增强处理；基于该第二鼻子图像对增强肖像A中的鼻子图像进行立体度增强处理；对增强肖像A中包含的眼神光的光斑轮廓清晰度和光斑亮度进行增强处理；获取虹膜信息库与增强肖像A的虹膜色彩信息的相似度较高的第一参考虹膜图像，基于第一参考虹膜图像对增强肖像A的眼睛部位的虹膜进行虹膜纹理和虹膜色彩的增强处理。Exemplarily, the above-mentioned second high-quality facial features image includes a second nose image, a second mouth image, and a second eyebrow image. Specifically, using the second high-quality facial features image to perform separate facial features enhancement processing on the enhanced portrait A may include but is not limited to: performing color and texture enhancement processing on the image of the lip area in the enhanced portrait A based on the second mouth image; performing color contrast enhancement processing on the image of the eyebrow area in the enhanced portrait A based on the second eyebrow image; performing stereoscopic enhancement processing on the nose image in the enhanced portrait A based on the second nose image; enhancing the spot contour clarity and spot brightness of the eye light contained in the enhanced portrait A; obtaining a first reference iris image with a high degree of similarity between the iris information library and the iris color information of the enhanced portrait A, and performing iris texture and iris color enhancement processing on the iris in the eye area of the enhanced portrait A based on the first reference iris image.
关于如何获取与第一图像匹配的第二优质五官图像，可以参照上述步骤S507中获取第一优质五官图像相关的类似说明，在此不再详述。Regarding how to obtain a second high-quality facial feature image that matches the first image, reference may be made to similar instructions related to obtaining the first high-quality facial feature image in the above step S507, which will not be described in detail here.
在一些可能的实现方式中，电子设备还可以对增强肖像A中包含的眼神光的光斑轮廓清晰度和光斑亮度进行增强处理；和/或，获取虹膜信息库与增强肖像A的虹膜色彩信息的相似度较高的第一参考虹膜图像，基于第一参考虹膜图像对增强肖像A的眼睛部位的虹膜进行虹膜纹理和虹膜色彩的增强处理。In some possible implementations, the electronic device may also enhance the spot contour clarity and spot brightness of the eye light contained in the enhanced portrait A; and/or obtain a first reference iris image having a high degree of similarity between the iris information library and the iris color information of the enhanced portrait A, and enhance the iris texture and iris color of the iris in the eye area of the enhanced portrait A based on the first reference iris image.
在本申请实施例中，电子设备基于第二优质五官图像对增强肖像A进行五官增强处理，以及增强肖像A中包含的眼神光的光斑轮廓清晰度和光斑亮度进行增强处理、基于第一参考虹膜图像对增强肖像A的眼睛部位的虹膜进行虹膜纹理和虹膜色彩的增强处理的功能，可以包含于上述步骤S608中所描述的肖像增强模型中，也可以是独立于该肖像增强模型之外的一个单独的增强模型，本文对此不做限定。In the embodiment of the present application, the electronic device performs facial feature enhancement processing on the enhanced portrait A based on the second high-quality facial feature image, enhances the spot contour clarity and spot brightness of the eye light contained in the enhanced portrait A, and enhances the iris texture and iris color of the iris in the eye area of the enhanced portrait A based on the first reference iris image. The function may be included in the portrait enhancement model described in the above step S608, or may be a separate enhancement model independent of the portrait enhancement model, which is not limited in this document.
在本申请实施例中，电子设备在执行完S616得到五官单独增强后的增强肖像A后，基于该五官单独增强后的增强肖像A执行步骤S609，也即对该五官单独增强后的增强肖像A和第一图像的背景图像进行背景融合增强得到第二图像。In an embodiment of the present application, after the electronic device executes S616 to obtain the enhanced portrait A after the facial features are individually enhanced, it executes step S609 based on the enhanced portrait A after the facial features are individually enhanced, that is, it performs background fusion enhancement on the enhanced portrait A after the facial features are individually enhanced and the background image of the first image to obtain the second image.
在一种可能的实现方式中，在上述图5和图7中，电子设备在确定第一图像满足增强条件之后，还可以基于电子设备的当前设备状态确定是否对第一图像进行数据解析以及图像增强处理。In a possible implementation, in the above-mentioned FIG. 5 and FIG. 7 , after the electronic device determines that the first image satisfies the enhancement condition, it may also determine whether to perform data analysis and image enhancement processing on the first image based on the current device state of the electronic device.
如图8所示，在步骤S502确定第一图像存在人脸之后，电子设备执行如下步骤：As shown in FIG8 , after determining in step S502 that a face exists in the first image, the electronic device performs the following steps:
S801，电子设备确定当前电量是否大于或等于预设电量阈值。S801, the electronic device determines whether the current power level is greater than or equal to a preset power threshold.
例如，预设电量阈值可以是电子设备的满电电量的30％或50％，本文对此不做限定。For example, the preset power threshold may be 30% or 50% of the full power of the electronic device, which is not limited herein.
若电子设备确定当前电量大于或等于预设电量阈值，则电子设备执行步骤S503至S512。若电子设备确定当前电量小于预设电量阈值，则执行步骤S802。If the electronic device determines that the current power level is greater than or equal to the preset power level threshold, the electronic device executes steps S503 to S512. If the electronic device determines that the current power level is less than the preset power level threshold, step S802 is executed.
S802，电子设备存储第一图像。S802: The electronic device stores the first image.
电子设备将第一图像作为历史图像存储到上述图像数据库中。例如，电子设备还可以在存储第一图像时，记录该第一图像属于人物图像。The electronic device stores the first image as a historical image in the image database. For example, when storing the first image, the electronic device may also record that the first image is a person image.
S803，电子设备确定当前是否处于充电灭屏状态且当前电量大于或等于预设电量阈值，或者，是否接收到用户主动触发的关于该第一图像的图像增强请求。S803, the electronic device determines whether it is currently in a charging and screen-off state and the current power level is greater than or equal to a preset power threshold, or whether it has received an image enhancement request for the first image actively triggered by the user.
在本申请实施例中，电子设备存储第一图像之后，若确定设备当前处于充电灭屏状态且当前电量大于或等于预设电量阈值，或者，接收到用户主动触发的关于该第一图像的图像增强请求，则电子设备对该第一图像执行步骤S503至S512。若电子设备当前既未处于充电灭屏状态或当前电量小于预设电量阈值、也未接收到用户主动触发的关于该第一图像的图像增强请求，则电子设备循环执行步骤S803，具体可以是周期性执行，本文对此不做限定。In the embodiment of the present application, after the electronic device stores the first image, if it is determined that the device is currently in the charging screen-off state and the current power is greater than or equal to the preset power threshold, or if an image enhancement request for the first image is received that is actively triggered by the user, the electronic device executes steps S503 to S512 for the first image. If the electronic device is currently neither in the charging screen-off state nor the current power is less than the preset power threshold, nor has an image enhancement request for the first image been actively triggered by the user, the electronic device executes step S803 cyclically, which may be performed periodically, and this document does not limit this.
例如，电子设备可以周期性地确定存储的人物图像中，是否存在属于人物图像的历史参考图像且电子设备中未存储有该历史参考图像的人物抠图数据、人脸解析数据、相关评分数据(例如五官评分结果、以及面部评分结果中的一项或多项)、人脸本征信息等数据。若电子设备确定存在该参考图像、且电子设备当前处于充电灭屏状态、以及电子设备的当前电量大于或等于预设电量阈值，则电子设备主动对该历史参考图像(其中这里的历史参考图像包括存储到图像数据库中的上述第一图像)执行步骤S503至S512。For example, the electronic device may periodically determine whether there is a historical reference image belonging to the character image in the stored character images and the character cutout data, face analysis data, related scoring data (such as one or more of the facial scoring results, and the facial scoring results) of the historical reference image are not stored in the electronic device, and the facial intrinsic information and other data. If the electronic device determines that the reference image exists, and the electronic device is currently in a charging and screen-off state, and the current power of the electronic device is greater than or equal to a preset power threshold, the electronic device actively performs steps S503 to S512 on the historical reference image (where the historical reference image here includes the above-mentioned first image stored in the image database).
例如，电子设备中的图库应用可以向用户展示图像数据库存储的图像，并向用户提供发起图像增强请求的入口，电子设备在接收到用户主动触发的关于该第一图像的增强请求后，无论电子设备的当前设备状态为何种状态，均对第一图像执行步骤S503至S512。For example, a gallery application in an electronic device can display images stored in an image database to a user and provide the user with an entry for initiating an image enhancement request. After the electronic device receives the enhancement request for the first image actively triggered by the user, the electronic device executes steps S503 to S512 on the first image regardless of the current device state of the electronic device.
可理解的，如图8所示的关于是否对第一图像进行数据解析以及图像增强处理的方案，也同样也适用于本申请图6所示的方法图像增强方法，在此不再详述。It is understandable that the solution on whether to perform data analysis and image enhancement processing on the first image as shown in FIG. 8 is also applicable to the image enhancement method shown in FIG. 6 of the present application, and will not be described in detail here.
在一种可能的实现方式中，若在拍照场景获取到的上述第一图像，电子设备可以基于图8所示的电量控制方法确定是否对第一图像采用本申请提供的方法进行图像增强。示例性的，对于拍照获取到的第一图像，若电子设备当前电量大于或等于上述预设电量阈值，则电子设备对第一图像进行图像增强；若电子设备确定当前电量小于上述预设电量阈值，则将第一图像作为历史图像存储。之后，若电子设备接收到用户主动请求对存储的第一图像进行图像处理的指令，则立即获取优质图像第一图像进行增强处理，优先满足用户需求。若电子设备未接收到用户主动请求对第一图像进行图像处理的指令，则电子设备可以在设备处于充电灭屏状态且当前电量大于或等于预设电量阈值的情况下，再采用本申请提供的图像增强方法对电子设备中存储的该第一图像进行图像增强处理，改善图像增强对设备的电能损耗问题。In a possible implementation, if the above-mentioned first image is acquired in the photo scene, the electronic device can determine whether to use the method provided in the present application to enhance the first image based on the power control method shown in FIG8. Exemplarily, for the first image acquired by taking a photo, if the current power of the electronic device is greater than or equal to the above-mentioned preset power threshold, the electronic device performs image enhancement on the first image; if the electronic device determines that the current power is less than the above-mentioned preset power threshold, the first image is stored as a historical image. Afterwards, if the electronic device receives an instruction from the user to actively request image processing of the stored first image, the high-quality first image is immediately acquired for enhancement processing, giving priority to meeting the user's needs. If the electronic device does not receive an instruction from the user to actively request image processing of the first image, the electronic device can use the image enhancement method provided in the present application to enhance the first image stored in the electronic device when the device is in a charging screen-off state and the current power is greater than or equal to the preset power threshold, thereby improving the problem of power consumption of the device due to image enhancement.
在另外一种可能的实现方式中，若电子设备基于数据下载方式在第一时间(例如第一时间为1分钟)内获取到至少两张第一图像，则电子设备在确定当前电量大于或等于预设电量阈值且设备当前处于充电灭屏状态的情况下，采用本申请提供的图像处理方法对第一图像进行图像增强；若电子设备当前电量小于预设电量阈值，则将获取到的上述至少两张第一图像作为历史图像存储。In another possible implementation, if the electronic device acquires at least two first images within a first time (for example, the first time is 1 minute) based on a data download method, the electronic device, when determining that the current power level is greater than or equal to a preset power threshold and the device is currently in a charging and screen-off state, uses the image processing method provided in the present application to enhance the first image; if the current power level of the electronic device is less than the preset power threshold, the at least two first images acquired are stored as historical images.
可理解的，数据传输例如蓝牙、NFC传输等就有一定的电能损耗，且若用户通过蓝牙或NFC等方式获取第一图像，也可以理解为当前用户对设备的使用需求较高，则基于本申请提供的方法，这种场景下电子设备需要满足电量大于或等于所述预设电量阈值且处于充电灭屏状态的情况，才能对获取到的人物图像进行增强处理，在达到图像增强效果的同时，改善图像增强为设备带来的电能损耗问题，以及为电子设备保持较好的电量状态以保障用户的设备使用需求。It is understandable that data transmission, such as Bluetooth, NFC transmission, etc., will have a certain amount of power consumption, and if the user obtains the first image through Bluetooth or NFC, etc., it can also be understood that the current user has a high demand for the use of the device. Based on the method provided in the present application, in this scenario, the electronic device needs to meet the condition that the power is greater than or equal to the preset power threshold and is in a charging and screen-off state before it can enhance the acquired character image. While achieving the image enhancement effect, it improves the power loss problem caused by image enhancement to the device, and maintains a good power state for the electronic device to ensure the user's device usage needs.
以下结合如图9所示的时序图，详细介绍在拍照场景中，电子设备中的图库应用如何采用本申请提供的图像处理方法对获得的图像进行增强处理。The following, in conjunction with the timing diagram shown in FIG. 9 , details how the gallery application in the electronic device uses the image processing method provided in the present application to enhance the obtained image in a photo-taking scenario.
如图9所示，在拍照场景中，电子设备执行本申请提供的图像处理方法所涉及的模块包括相机应用、图库应用、判断模块、人像处理模块、评分模块、人脸属性估计模块、增强融合模块、以及图像数据库。As shown in Figure 9, in the photo-taking scenario, the modules involved in the electronic device executing the image processing method provided in this application include a camera application, a gallery application, a judgment module, a portrait processing module, a scoring module, a face attribute estimation module, an enhanced fusion module, and an image database.
阶段1：人脸检测。Stage 1: Face detection.
S901，相机应用基于用户操作拍摄获取第一图像，并向图库应用发送该第一图像，相应地，图库应用接收该第一图像。S901, the camera application captures a first image based on a user operation, and sends the first image to the gallery application. Accordingly, the gallery application receives the first image.
例如，电子设备基于用户操作启用相机应用，并在接收到拍摄指令后拍摄得到第一图像。For example, the electronic device enables a camera application based on a user operation, and captures a first image after receiving a capture instruction.
S902，图库应用调用人像处理模块对第一图像进行人脸检测，获得人脸检测结果。S902: The gallery application calls the portrait processing module to perform face detection on the first image to obtain a face detection result.
人像处理模块对第一图像进行人脸检测，并向图库应用发送人脸检测结果，相应地，图库应用接收该人脸检测结果。The portrait processing module performs face detection on the first image and sends the face detection result to the gallery application. Correspondingly, the gallery application receives the face detection result.
若人脸检测结果指示第一图像中包含人脸图像，则电子设备执行步骤S903、S904、以及S905。若人脸检测结果指示第一图像中不包含人脸图像，则电子设备不对第一图像进行人脸数据解析以及图像增强处理。If the face detection result indicates that the first image contains a face image, the electronic device executes steps S903, S904, and S905. If the face detection result indicates that the first image does not contain a face image, the electronic device does not perform face data analysis and image enhancement processing on the first image.
在一种可能的实现方式中，图库应用在确定第一图像中包含人脸图像后，还可以将该第一图像存储到图像数据库中，并为该第一图像标上人物图像的标签，或者，人像处理模型确定第一图像包含人脸后，将第一图像存储到图形数据库中，并为该第一图像标上人物图像的标签。In one possible implementation, after determining that the first image contains a face image, the gallery application may store the first image in an image database and label the first image as a person image; or, after the portrait processing model determines that the first image contains a face, the first image may be stored in a graphic database and labeled as a person image.
若图库应用确定第一图像中不包含人脸图像，也可以将该第一图像存储到该图像数据库中，或者将该第一图像存储到除了该图像数据库之外的另外一个数据库(例如称数据库A)，该数据库A用于存储不包含人脸的图像，本文对此不做限定。If the gallery application determines that the first image does not contain a face image, the first image can also be stored in the image database, or the first image can be stored in another database other than the image database (for example, database A), where database A is used to store images that do not contain faces, which is not limited in this article.
在本申请实施例中，图像数据库可以存储在电子设备的本地存储空间中，也可以存储在电子设备的云服务器端，本文对此不做限定。In the embodiment of the present application, the image database can be stored in the local storage space of the electronic device, or stored in the cloud server of the electronic device, which is not limited in this document.
阶段2：获取图像数据并确定图像增强任务的执行时机。Phase 2: Acquire image data and determine when to perform image enhancement tasks.
S903，图库应用调用人像处理模块对第一图像进行人脸聚类、人像抠图、人脸解析、以及人脸特征提取。S903, the gallery application calls the portrait processing module to perform face clustering, portrait cutout, face analysis, and face feature extraction on the first image.
人脸抠图数据包括第一图像的肖像和背景图像，人脸解析数据包括但不限于第一图像的待增强五官图像、待增强面部图像，人脸特征数据包括但不限于：人脸关键点、人脸纹理特征。在一些可能的实现方式中，人脸解析数据还可以包括人像的头发、脖子、以及饰品对应的图像。The face cutout data includes the portrait and background image of the first image, the face analysis data includes but is not limited to the facial features to be enhanced and the face image to be enhanced of the first image, and the face feature data includes but is not limited to: facial key points and facial texture features. In some possible implementations, the face analysis data may also include images corresponding to the hair, neck, and accessories of the portrait.
在本申请实施例中，人像处理模块获取到第一图像的聚类标签、人像抠图数据、人脸解析数据、以及人脸特征数据后，将其存储到图像数据库中。In the embodiment of the present application, after the portrait processing module obtains the clustering label, portrait cutout data, face analysis data, and face feature data of the first image, it stores them in the image database.
S904，图库应用调用人脸属性估计模块对第一图像进行头部姿态估计。S904: The gallery application calls a face attribute estimation module to perform head posture estimation on the first image.
关于第一图像的头部姿态以及人脸本征信息等的说明可以参照上文其他实施例的相关说明，例如上述步骤S505中的相关说明，在此不再详述。For the description of the head posture and facial intrinsic information of the first image, reference may be made to the relevant description of other embodiments above, such as the relevant description in the above step S505, which will not be described in detail here.
在本申请实施例中，人脸属性估计模块获取到将第一图像的头部姿态后，将其存储到图像数据库中。In the embodiment of the present application, after the face attribute estimation module obtains the head posture of the first image, it stores it in the image database.
S905，图库应用调用判断模块确定第一图像是否满足增强条件，获得判断结果。S905: The gallery application calls a determination module to determine whether the first image meets an enhancement condition and obtains a determination result.
关于增强条件、以及判断模块具体如何判断第一图像是否满足增强条件可以参照上文相关说明，例如步骤S506中的相关说明。Regarding the enhancement condition and how the determination module determines whether the first image satisfies the enhancement condition, reference may be made to the above related description, such as the related description in step S506.
在确定判断结果为第一图像满足增强条件的情况下，执行步骤S906。在确定判断结果为第一图像不满足增强条件的情况下，不对第一图像进行图像增强处理。If the judgment result is that the first image meets the enhancement condition, step S906 is performed. If the judgment result is that the first image does not meet the enhancement condition, no image enhancement processing is performed on the first image.
S906，图库应用调用评分模块对第一图像以及图像数据库中未评分的人物图像进行图像评分。S906: The gallery application calls a scoring module to score the first image and unrated person images in the image database.
上述图像评分可以包括画质评分、面部评分以及五官评分。The above-mentioned image rating may include an image quality rating, a face rating, and a facial features rating.
示例性的，评分模块可以获取图像数据库中存储的相关人脸数据，并根据图像数据库返回的相关人脸数据进行图像评分。Exemplarily, the scoring module may obtain relevant face data stored in an image database, and perform image scoring based on the relevant face data returned by the image database.
例如，图像评分包含画质评分、面部评分以及五官评分，则评分模块向图像数据库获取第一图像的肖像、面部图像以及五官图像(也可以是第一图像对应的人像抠图数据和人脸解析数据)、以及获取图像数据库中除了该第一图像之外的其他未评分的人物图像的肖像、面部图像、五官图像(也可以是该未评分人物图像对应的人像抠图数据和人脸解析数据)。评分模块获取到第一图像的面部图像和五官图像、以及获取图像数据库中未评分的人物图像的面部图像和五官图像后，相应地进行画质评分、面部评分以及五官评分，并将人物图像对应的画质评分结果、面部评分结果以及五官评分结果存储到图像数据库中。For example, if the image scoring includes image quality scoring, facial scoring, and facial features scoring, the scoring module obtains the portrait, facial image, and facial features image of the first image (or the portrait cutout data and face analysis data corresponding to the first image) from the image database, and obtains the portrait, facial image, and facial features image of other unscored person images other than the first image from the image database (or the portrait cutout data and face analysis data corresponding to the unscored person images). After the scoring module obtains the facial image and facial features image of the first image, and the facial image and facial features image of the unscored person images in the image database, it performs image quality scoring, facial scoring, and facial features scoring accordingly, and stores the image quality scoring results, facial scoring results, and facial features scoring results corresponding to the person images in the image database.
具体如何对图像进行画质评分、五官评分、或面部评分，可以参照本文其他相关说明，例如上述步骤S504或上述步骤S604，在此不再详述。For specific methods of performing image quality scoring, facial feature scoring, or face scoring on an image, please refer to other relevant instructions in this document, such as the above-mentioned step S504 or the above-mentioned step S604, which will not be described in detail here.
在一种可能的实现方式中，也可以由判断模块调用评分模块执行上述步骤S906，也即判断模块在确定第一图像满足增强条件的情况下，调用评分模块对第一图像以及图像数据库中未评分的图像进行面部评分、以及五官评分，本文对此不做限定。In a possible implementation, the judgment module may also call the scoring module to execute the above step S906, that is, when the judgment module determines that the first image meets the enhancement conditions, the scoring module is called to perform facial scoring and facial feature scoring on the first image and the unscored images in the image database, which is not limited in this document.
阶段3：确定图像增强任务的执行时机并对第一图像进行图像增强。Stage 3: Determine the execution timing of the image enhancement task and perform image enhancement on the first image.
S907，图库应用确定图像数据库中人物图像的评分数据是否完善。S907 , the gallery application determines whether the rating data of the person image in the image database is complete.
在本申请实施例中，人物图像的评分数据是指图像数据库中存储的除了已进行过图像增强之外的全部人物图像的评分数据，该评分数据包含人物图像的画质评分结果、面部评分结果和五官评分结果。In an embodiment of the present application, the scoring data of a person image refers to the scoring data of all person images stored in the image database except those that have undergone image enhancement, and the scoring data includes the image quality scoring results, facial scoring results, and facial features scoring results of the person images.
例如，评分数据是指人物图像的画质评分结果、面部评分结果以及五官评分结果，则若图像数据库中已存储有所有人物图像(包括第一图像)的画质评分结果、面部评分结果和五官评分结果，则人物图像的评分数据完善。也可以说明图像数据库中已存储有所有人物图像(包括第一图像)的肖像、面部图像、五官图像、以及头部姿态。For example, the scoring data refers to the image quality scoring results, face scoring results, and facial features scoring results of the person image. If the image database has stored the image quality scoring results, face scoring results, and facial features scoring results of all person images (including the first image), the scoring data of the person image is complete. It can also be explained that the image database has stored the portraits, face images, facial features images, and head postures of all person images (including the first image).
在本申请实施例中，图库应用可以周期性查询图像数据库中人物图像的评分数据是否完善。In an embodiment of the present application, the gallery application may periodically query whether the rating data of the person images in the image database is complete.
可理解的，如果图库应用确定第一图像满足增强条件之后，不确认图像数据库中人物图像的评分数据是否完善就立即调用增强融合模块对第一图像进行人脸增强，就可能存在图像数据库中存储的人物图像的数据缺失，无法向增强融合模块提供对第一图像进行人脸增强所需的数据的问题，出现无法预知的代码运行错误(bug)，降低程序代码的成功率。It is understandable that if the gallery application determines that the first image meets the enhancement conditions, but immediately calls the enhancement fusion module to perform face enhancement on the first image without confirming whether the scoring data of the person images in the image database is complete, there may be a problem that the data of the person images stored in the image database is missing, and the data required for face enhancement of the first image cannot be provided to the enhancement fusion module, resulting in unpredictable code execution errors (bugs), reducing the success rate of the program code.
S908，图库应用在确定第一图像满足增强条件、且图像数据库中图像的评分数据完善的情况下，调用增强融合模块对第一图像进行图像增强处理。S908: When the gallery application determines that the first image meets the enhancement condition and the image rating data in the image database is complete, the image library application calls the enhancement fusion module to perform image enhancement processing on the first image.
可理解的，增强融合模块可以基于图像数据库中存储的人物图像的相关数据(包括人物图像的聚类标签、人像抠图数据、人脸解析数据、人脸特征数据、头部姿态、以及图像评分数据)，通过图5或图6所示的任意一种图像处理方法对第一图像进行图像增强处理，得到上述第二图像。例如可以基于步骤S507至S512对第一图像进行图像增强处理。也可以基于步骤S607至S609对第一图像进行图像增强处理，或者，也可以基于S607、S610至S612、以及S609对第一图像进行图像增强处理，或者也可以基于S607至S609以及S615至S616对第一图像进行图像增强处理，或者，也可以基于S607至S616对第一图像进行图像增强处理，还可以基于图7所示的图像处理方法对第一图像进行图像增强处理。It is understandable that the enhancement fusion module can perform image enhancement processing on the first image based on the relevant data of the character image stored in the image database (including clustering labels of the character image, portrait cutout data, face analysis data, face feature data, head posture, and image scoring data) through any image processing method shown in Figure 5 or Figure 6 to obtain the above-mentioned second image. For example, the first image can be image enhanced based on steps S507 to S512. The first image can also be image enhanced based on steps S607 to S609, or the first image can be image enhanced based on S607, S610 to S612, and S609, or the first image can be image enhanced based on S607 to S609 and S615 to S616, or the first image can be image enhanced based on S607 to S616, or the first image can be image enhanced based on the image processing method shown in Figure 7.
在本申请实施例中，增强融合模块在获取到第二图像后，将第二图像存储到图像数据库中。例如，增强融合模块删除原始的第一图像，将第二图像存储到图像数据库中。In the embodiment of the present application, after acquiring the second image, the enhancement fusion module stores the second image in the image database. For example, the enhancement fusion module deletes the original first image and stores the second image in the image database.
在另外一种可能的实现方式中，电子设备在接收到用户主动触发的关于上述第一图像的增强请求且该第一图像包含人脸图像的情况下，电子设备也可以基于上述图9所示的S902至S908所示的图像增强方法对第一图像进行图像增强处理。In another possible implementation, when the electronic device receives an enhancement request for the first image actively triggered by the user and the first image contains a facial image, the electronic device may also perform image enhancement processing on the first image based on the image enhancement method shown in S902 to S908 shown in Figure 9 above.
在本申请实施例中，如图8所示的基于当前设备状态确定是否对第一图像进行数据解析以及图像增强处理的方法也同样适用于图9所示的方法流程图，在此不再详述。例如，在步骤S902确定第一图像存在人脸之后，执行S801，并在确定电子设备的当前电量大于或等于预设电量阈值的情况下，执行步骤S903至S908。以及，在确定电子设备的当前电量小于预设电量阈值的情况下执行步骤S802至S803，其中在S802中对第一图像执行步骤S903和S904，在确定电子设备处于充电灭屏状态或接收到用户发起的针对该第一图像的图像增强请求后执行步骤S905至S908。In an embodiment of the present application, the method for determining whether to perform data analysis and image enhancement processing on the first image based on the current device state as shown in FIG8 is also applicable to the method flow chart shown in FIG9, which will not be described in detail here. For example, after determining in step S902 that there is a face in the first image, S801 is executed, and when it is determined that the current power of the electronic device is greater than or equal to the preset power threshold, steps S903 to S908 are executed. And, when it is determined that the current power of the electronic device is less than the preset power threshold, steps S802 to S803 are executed, wherein steps S903 and S904 are executed on the first image in S802, and steps S905 to S908 are executed after determining that the electronic device is in a charging screen-off state or receiving an image enhancement request initiated by the user for the first image.
示例性的，在一些场景中，电子设备在接收到用户的拍摄指令后，获取包含人脸的上述第一图像并存储到图库中，在电子设备满足相应的设备状态且第一图像满足增强条件的情况下，对第一图像进行增强处理，并在接收到关于该第一图像的显示操作后，显示增强处理后的上述第二图像。Exemplarily, in some scenarios, after receiving a shooting instruction from the user, the electronic device obtains the above-mentioned first image containing a face and stores it in a gallery. When the electronic device meets the corresponding device state and the first image meets the enhancement condition, the first image is enhanced. After receiving a display operation on the first image, the above-mentioned second image after enhancement is displayed.
例如，如图10所示，电子设备通过相机应用拍照获取图像A，该图像A中包含肖像图像；具体的，如图10用户界面10A所示，电子设备在接收到关于拍照控件1001的点击操作后，拍摄得到该图像A，将拍摄到的图像A存储到图库应用中，并如图10用户界面10B所示在图像预览区域1002显示该图像A的缩小图。For example, as shown in FIG10 , the electronic device obtains image A by photographing through a camera application, and image A includes a portrait image; specifically, as shown in user interface 10A of FIG10 , after receiving a click operation on a photographing control 1001, the electronic device photographs image A, stores the photographed image A in a gallery application, and displays a reduced image of image A in image preview area 1002 as shown in user interface 10B of FIG10 .
示例性的，电子设备在拍摄获得上述图像A后，确定当前电量是否大于或等于预设电量阈值。若电子设备的当前电量大于或等于预设电量阈值，则电子设备基于上述步骤S903至S908在图像A满足增强条件的情况下，对图像A进行图像增强处理，得到与图像A对应的增强好的图像B，并如用户界面10C所示，将显示图像A替换为显示该图像B，该图像B的图像质量优于图像A。Exemplarily, after capturing and obtaining the above-mentioned image A, the electronic device determines whether the current power is greater than or equal to the preset power threshold. If the current power of the electronic device is greater than or equal to the preset power threshold, the electronic device performs image enhancement processing on image A based on the above-mentioned steps S903 to S908 when image A meets the enhancement condition, obtains an enhanced image B corresponding to image A, and replaces the displayed image A with the displayed image B, as shown in the user interface 10C, and the image quality of image B is better than that of image A.
示例性的，若用户当前通过电子设备在户外拍摄M(M大于或等于1)张人物图像，电子设备在确定当前电量大于或等于预设电量阈值时，采用本申请提供的任一种图像处理方法对该M张人物图像进行图像增强，当处理到第X张(X小于或等于M)人物图像后发现电子设备的当前电量小于预设电量阈值时，则停止对其他人物图像进行增强处理，从而可以为电子设备保持相当的电量以保障用户的户外活动的设备需求。Exemplarily, if a user currently uses an electronic device to take M (M is greater than or equal to 1) images of people outdoors, when the electronic device determines that the current power level is greater than or equal to a preset power threshold, it uses any image processing method provided in the present application to enhance the M images of people. When after processing the Xth (X is less than or equal to M) image of a person and finding that the current power level of the electronic device is less than the preset power threshold, it stops enhancing other images of the person, thereby maintaining a sufficient power level for the electronic device to ensure the equipment needs of the user's outdoor activities.
在一种可能的实现方式中，若在电子设备完成对图像A的增强已获得图像B之后，接收到查看该图像A的指令(例如如图10用户界面10B所示，电子设备接收到关于上述图像预览区域1002的点击操作)，则电子设备显示增强好的图像B。若在电子设备对图像A进行图像增强以获得图像B的过程中，接收到关于图像预览区域1002的点击操作，则电子设备可以先显示图像A，并输出图像增强中的提示信息(例如该提示信息为提示正在加载中的任意形式的提示信息)，以及在获得增强好的图像B之后，将显示图像A替换为显示该图像B。In a possible implementation, if after the electronic device completes the enhancement of image A to obtain image B, an instruction to view the image A is received (for example, as shown in user interface 10B of FIG. 10 , the electronic device receives a click operation on the image preview area 1002), the electronic device displays the enhanced image B. If during the process of the electronic device performing image enhancement on image A to obtain image B, a click operation on the image preview area 1002 is received, the electronic device may first display image A, and output prompt information during image enhancement (for example, the prompt information is any form of prompt information indicating that the image is being loaded), and after obtaining the enhanced image B, the display of image A is replaced by the display of image B.
示例性的，如图10中的图像A和图像B的效果图所示，图像A满足增强条件中的画质模糊条件，采用本申请提供的图像处理方法对该第一图像进行处理后，得到图像B。其中，该图像B中的面部区域的清晰度更高、纹理细节更多(具体可以是基于优质肖像对待增强肖像进行增强得到，也可以是基于优质面部图像对待增强面部图像进行增强得到)，五官区域的清晰度更高、嘴唇纹理细节更丰富、眉毛毛流感更佳、鼻子立体感更佳、眼睛中眼神光的光斑轮廓的更好清晰度以及光斑亮度更好、眼睛中虹膜纹理细节和色彩信息更好(具体可以是基于优质肖像对待增强肖像进行增强得到，也可以是基于优质五官图像对待增强五官图像进行增强得到)，图像的整体画质清晰度更佳，例如衣服区域的清晰度更高(例如可以是基于优质肖像对待增强肖像进行增强得到)，例如原始背景图像区域的清晰度更高(具体可以是基于增强后的肖像对原始背景图像进行色彩校正得到)，脖子区域的清晰度更高、纹理细节更多(具体可以是对脖子区域单独进行清晰度和纹理增强处理得到)，头发区域的清晰度和光泽感更好(具体可以是对头发区域单独进行清晰度、对比度、光泽度增强处理得到)。Exemplarily, as shown in the effect diagram of image A and image B in FIG10 , image A satisfies the image quality blur condition in the enhancement condition, and after the first image is processed by the image processing method provided in the present application, image B is obtained. Among them, the facial area in the image B has higher clarity and more texture details (specifically, it can be obtained by enhancing the enhanced portrait based on the high-quality portrait, or by enhancing the enhanced facial image based on the high-quality facial image), the facial features area has higher clarity, the lip texture details are richer, the eyebrow hair flow is better, the nose has better three-dimensional sense, the spot contour of the eye light in the eye is better in clarity and the spot brightness is better, the iris texture details and color information in the eye are better (specifically, it can be obtained by enhancing the enhanced portrait based on the high-quality portrait, or by enhancing the enhanced facial features image based on the high-quality facial features image), the overall image quality clarity is better, for example, the clarity of the clothes area is higher (for example, it can be obtained by enhancing the enhanced portrait based on the high-quality portrait), for example, the clarity of the original background image area is higher (specifically, it can be obtained by color correcting the original background image based on the enhanced portrait), the clarity of the neck area is higher and the texture details are more (specifically, it can be obtained by performing clarity and texture enhancement processing on the neck area alone), and the clarity and gloss of the hair area are better (specifically, it can be obtained by performing clarity, contrast, and gloss enhancement processing on the hair area alone).
在本申请实施例中，电子设备可以通过图库应用向用户提供手动触发对图像A进行增强的功能，也就是说用户可以通过图库应用手动触发请求对图像A进行增强。In an embodiment of the present application, the electronic device can provide the user with a function of manually triggering enhancement of image A through a gallery application, that is, the user can manually trigger a request to enhance image A through the gallery application.
示例性的，电子设备中存储有上述图像A，且该图像A未进行过图像增强处理。电子设备基于用户操作通过图库应用显示如图11所示的用户界面11A，该用户界面11A包含该图像A以及工具栏，工具栏中包含人像增强控件1101，该人像增强控件1101即为电子设备通过图库应用向用户提供的手动触发对图像A进行增强的功能。电子设备在接收到关于该人像增强控件1101的点击操作后，基于上述步骤S903至S908对该图像A进行图像增强，并如图11用户界面11B所示，显示增强好的照片，也即显示图像A对应的增强好的图像B。具体的，电子设备对该图像A进行图像增强得到该图像B后，利用该图像B替换该图像A存储到图库应用中。Exemplarily, the electronic device stores the above-mentioned image A, and the image A has not been subjected to image enhancement processing. Based on the user operation, the electronic device displays the user interface 11A shown in FIG11 through the gallery application. The user interface 11A includes the image A and a toolbar. The toolbar includes a portrait enhancement control 1101. The portrait enhancement control 1101 is a function for manually triggering the enhancement of image A provided by the electronic device to the user through the gallery application. After receiving the click operation on the portrait enhancement control 1101, the electronic device performs image enhancement on the image A based on the above steps S903 to S908, and displays the enhanced photo as shown in the user interface 11B of FIG11, that is, displays the enhanced image B corresponding to the image A. Specifically, after the electronic device performs image enhancement on the image A to obtain the image B, it replaces the image A with the image B and stores it in the gallery application.
可理解的，工具栏还可以包含编辑控件、收藏控件、删除控件等，本文对此不做限定。It is understandable that the toolbar may also include edit controls, favorite controls, delete controls, etc., which are not limited in this article.
在一些可能的实现方式中，电子设备通过图库应用显示一张参考照片时，对应的显示界面中的工具栏中是否包含上述人像增强控件1101可以根据该参考照片是否包含肖像以及该参考照片是否满足增强条件确定。示例性的，若参考照片包含肖像且该参考照片满足增强条件，则该参考图像对应的显示界面中的工具栏中包含人像增强控件1101，若参考照片不包含肖像或参考照片不满足增强条件，则该参考图像对应的显示界面中的工具栏中可以不包含人像增强控件1101。In some possible implementations, when an electronic device displays a reference photo through a gallery application, whether the toolbar in the corresponding display interface includes the above-mentioned portrait enhancement control 1101 can be determined based on whether the reference photo includes a portrait and whether the reference photo meets the enhancement condition. Exemplarily, if the reference photo includes a portrait and the reference photo meets the enhancement condition, the toolbar in the display interface corresponding to the reference image includes the portrait enhancement control 1101; if the reference photo does not include a portrait or the reference photo does not meet the enhancement condition, the toolbar in the display interface corresponding to the reference image may not include the portrait enhancement control 1101.
在另外一些可能的场景中，电子设备也可以基于数据下载的方式(例如网页下载、网盘下载、蓝牙数据传输、近场通信NFC等数据传输方式)在短时间内获取到至少一张上述第一图像。In some other possible scenarios, the electronic device may also acquire at least one of the above-mentioned first images in a short time based on data downloading methods (such as web page downloading, network disk downloading, Bluetooth data transmission, near field communication NFC and other data transmission methods).
在这种场景中，若电子设备在短时间内通过数据下载的方式接收到大量的第一图像(例如接收到的第一图像的数量大于或等于两张)，则电子设备可以基于当前设备状态确定是否对该大量的第一图像进行数据解析以及图像增强处理。In this scenario, if the electronic device receives a large number of first images (for example, the number of received first images is greater than or equal to two) through data download in a short period of time, the electronic device can determine whether to perform data analysis and image enhancement processing on the large number of first images based on the current device status.
具体的，电子设备确定设备当前是否处于充电灭屏状态以及当前电量是否大于或等于预设电量阈值，若电子设备当前处于充电灭屏状态且当前电量大于或等于预设电量阈值，则基于上述步骤S902至S908对接收到的上述大量的第一图像进行数据解析和图像增强处理。若电子设备当前未处于充电灭屏状态，或当前电量小于预设电量阈值，则电子设备将该大量的图像存储到图像数据库中，并在电子设备处于充电灭屏状态且当前电量大于或等于预设电量阈值后，电子设备对图像数据库中评分数据未完善的图像(也即包含上述大量的图像)通过上述步骤S902至S908所示的图像增强方法进行数据解析和图像增强处理。Specifically, the electronic device determines whether the device is currently in a charging screen-off state and whether the current power level is greater than or equal to a preset power threshold. If the electronic device is currently in a charging screen-off state and the current power level is greater than or equal to the preset power threshold, then the received large number of first images are subjected to data analysis and image enhancement processing based on the above steps S902 to S908. If the electronic device is not currently in a charging screen-off state, or the current power level is less than the preset power threshold, the electronic device stores the large number of images in an image database, and after the electronic device is in a charging screen-off state and the current power level is greater than or equal to the preset power threshold, the electronic device performs data analysis and image enhancement processing on images in the image database whose scoring data is incomplete (i.e., including the large number of images) through the image enhancement method shown in the above steps S902 to S908.
在另外一种可能的实现方式中，若电子设备在短时间内通过数据传输的方式接收到少量的图像(例如接收到的图像的数量为1)，则电子设备可以基于上述步骤S902至S908所示的图像增强方法对接收到的图像进行图像增强处理。或者，若电子设备在短时间内通过数据传输的方式接收到少量的图像、且电子设备确定当前电量大于或等于上述预设电量阈值的情况下，基于上述步骤S902至S908所示的图像增强方法对接收到的图像进行图像增强处理。或者，若电子设备在短时间内通过数据传输的方式接收到少量的图像、且电子设备确定当前电量小于预设电量阈值的情况下，电子设备存储该少量的图像，并在电子设备处于充电灭屏状态后，基于上述步骤S902至S908所示的图像增强方法对接收到的图像进行图像增强处理。In another possible implementation, if the electronic device receives a small amount of images (for example, the number of received images is 1) by means of data transmission in a short time, the electronic device may perform image enhancement processing on the received images based on the image enhancement method shown in steps S902 to S908. Alternatively, if the electronic device receives a small amount of images by means of data transmission in a short time, and the electronic device determines that the current power is greater than or equal to the preset power threshold, the image enhancement processing is performed on the received images based on the image enhancement method shown in steps S902 to S908. Alternatively, if the electronic device receives a small amount of images by means of data transmission in a short time, and the electronic device determines that the current power is less than the preset power threshold, the electronic device stores the small amount of images, and after the electronic device is in the charging screen-off state, the image enhancement processing is performed on the received images based on the image enhancement method shown in steps S902 to S908.
在本申请实施例中，上述实施例1或实施例2中图像数据库中存储的人物图像的面部图像、五官图像、肖像、背景图像，可以以人像抠图mask和人脸解析mask的方式存储，例如如图1所示，直接存储人物图像的人像抠图语义分割mask，和103至108所示的人脸解析语义分割mask。或者，电子设备也可以是以单份图像数据的方式存储面部图像、五官图像、肖像、以及背景图像，例如如图1所示，以单独的一份图像数据的方式存储人物图像中肖像101和背景图像102，以及以单独的一份图像数据的方式存储103至108中的每一项图像，根据具体设计和需求而定，本文对此不做限定。In an embodiment of the present application, the facial image, facial features image, portrait, and background image of the person image stored in the image database in the above-mentioned embodiment 1 or embodiment 2 can be stored in the form of a portrait cutout mask and a face analysis mask. For example, as shown in FIG1 , the portrait cutout semantic segmentation mask of the person image and the face analysis semantic segmentation mask shown in 103 to 108 are directly stored. Alternatively, the electronic device can also store the facial image, facial features image, portrait, and background image in the form of a single image data. For example, as shown in FIG1 , the portrait 101 and the background image 102 in the person image are stored in the form of a separate image data, and each of the images 103 to 108 are stored in the form of a separate image data, according to the specific design and requirements, which is not limited in this document.
在一些可能的实现方式中，第一图像中包含的肖像的数量大于或等于2，则电子设备可以分别对每一个肖像进行图像增强处理。示例性的，第一图像包含肖像A和肖像B，则电子设备可以分别将该肖像A和肖像B作为上述图6所示的待增强肖像，分别利用与肖像A对应的优质肖像对肖像A、利用肖像B对应的优质肖像对肖像B进行增强处理。或者，电子设备也可以分别将该肖像A的五官图像和肖像B的五官图像分别作为图5所示的待增强五官图像，分别利用与该肖像A的五官图像对应的优质五官图像对该肖像A的五官图像进行增强处理、利用与该肖像B的五官图像对应的优质五官图像对该肖像B的五官图像进行增强处理，以及将肖像A的面部图像和肖像B的面部图像分别作为图5所示的待增强面部图像，分别利用与该肖像A的面部图像对应的优质面部图像对该肖像A的面部图像进行增强处理、利用与该肖像B的面部图像对应的优质面部图像对该肖像B的面部图像进行增强处理。In some possible implementations, if the number of portraits included in the first image is greater than or equal to 2, the electronic device may perform image enhancement processing on each portrait respectively. Exemplarily, if the first image includes portrait A and portrait B, the electronic device may respectively use the portrait A and portrait B as the portraits to be enhanced as shown in FIG. 6, and respectively use the high-quality portrait corresponding to portrait A to enhance portrait A, and use the high-quality portrait corresponding to portrait B to enhance portrait B. Alternatively, the electronic device may also respectively use the facial features image of portrait A and the facial features image of portrait B as the facial features images to be enhanced as shown in FIG. 5, and respectively use the high-quality facial features image corresponding to the facial features image of portrait A to enhance the facial features image of portrait A, and use the high-quality facial features image corresponding to the facial features image of portrait B to enhance the facial features image of portrait B, and use the facial image of portrait A and the facial image of portrait B as the facial images to be enhanced as shown in FIG. 5, and respectively use the high-quality facial image corresponding to the facial image of portrait A to enhance the facial image of portrait A, and use the high-quality facial image corresponding to the facial image of portrait B to enhance the facial image of portrait B.
可理解的，可以由任意具备运行图像处理算法和模型的电子设备执行本申请实施例提供的图像增强方法。该电子设备的形态可以包括但不限于非折叠屏手机终端、折叠屏手机终端、平板终端、笔记本电脑、桌面型计算机、膝上型计算机、或手持计算机等，本文对此不做限定。It is understandable that the image enhancement method provided in the embodiment of the present application can be performed by any electronic device capable of running image processing algorithms and models. The form of the electronic device may include but is not limited to a non-folding screen mobile phone terminal, a folding screen mobile phone terminal, a tablet terminal, a laptop computer, a desktop computer, a laptop computer, or a handheld computer, etc., which is not limited in this article.
请参阅图12，以图12所示的电子设备100为例，详细说明本申请提供的电子设备的结构示意图。Please refer to FIG. 12 , which takes the electronic device 100 shown in FIG. 12 as an example to describe in detail the structural diagram of the electronic device provided in the present application.
电子设备100可以包括处理器110，外部存储器120，内部存储区121，通用串行总线(universal serial bus，USB)接口130，充电管理模块140，电源管理模块141，电池142，天线1，天线2，移动通信模块150，无线通信模块160，传感器模块170，按键180，摄像头190，以及显示屏191。其中传感器模块170可以包括压力传感器，陀螺仪传感器，气压传感器，磁传感器，加速度传感器，距离传感器，接近光传感器，指纹传感器，温度传感器，触摸传感器，环境光传感器，骨传导传感器等。The electronic device 100 may include a processor 110, an external memory 120, an internal storage area 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, a sensor module 170, a button 180, a camera 190, and a display screen 191. The sensor module 170 may include a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, and the like.
可以理解的是，本发明实施例示意的结构并不构成对电子设备100的具体限定。在本申请另一些实施例中，电子设备100可以包括比图12所示中更多或更少的部件，或者组合某些部件，或者拆分某些部件，或者不同的部件布置。图12所示的部件可以以硬件，软件或软件和硬件的组合实现。例如，电子设备100还可以包括音频模块(扬声器、受话器、麦克风、耳机接口等)、用户标识模块(subscriber identification module，SIM)卡接口、以及马达等。It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than those shown in FIG. 12, or combine certain components, or split certain components, or arrange the components differently. The components shown in FIG. 12 may be implemented in hardware, software, or a combination of software and hardware. For example, the electronic device 100 may also include an audio module (speaker, receiver, microphone, headphone jack, etc.), a subscriber identification module (SIM) card interface, and a motor, etc.
处理器110可以包括一个或多个处理单元，例如：处理器110可以包括应用处理器(application processor，AP)，调制解调处理器，图形处理器(graphics processingunit，GPU)，图像信号处理器(image signal processor，ISP)，控制器，存储器，视频编解码器，数字信号处理器(digital signal processor，DSP)，基带处理器，和/或神经网络处理器(neural-network processing unit，NPU)等。其中，不同的处理单元可以是独立的器件，也可以集成在一个或多个处理器中。The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and/or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.
示例性的，处理器110可以用于执行图4-图9任一项所示的图像处理方法，或者，处理器110也可以结合上述外部存储器120，内部存储区121，充电管理模块140，电源管理模块141、无线通信模块160，传感器模块170，按键180，摄像头190，以及显示屏191中的一项或多项，配合执行行图4-图9任一项所示的图像处理方法。Exemplarily, the processor 110 can be used to execute the image processing method shown in any one of Figures 4-9, or the processor 110 can also be combined with one or more of the above-mentioned external memory 120, internal storage area 121, charging management module 140, power management module 141, wireless communication module 160, sensor module 170, button 180, camera 190, and display screen 191 to cooperate in executing the image processing method shown in any one of Figures 4-9.
其中，控制器可以是电子设备100的神经中枢和指挥中心。控制器可以根据指令操作码和时序信号，产生操作控制信号，完成取指令和执行指令的控制。The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.
处理器110中还可以设置存储器，用于存储指令和数据。在一些实施例中，处理器110中的存储器为高速缓冲存储器。该存储器可以保存处理器110刚用过或循环使用的指令或数据。如果处理器110需要再次使用该指令或数据，可从所述存储器中直接调用。避免了重复存取，减少了处理器110的等待时间，因而提高了系统的效率。The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.
在一些实施例中，处理器110可以包括一个或多个接口。接口可以包括集成电路(inter-integrated circuit，I2C)接口，集成电路内置音频(inter-integrated circuitsound，I2S)接口，脉冲编码调制(pulse code modulation，PCM)接口，通用异不收发传输器(universal asynchronous receiver/transmitter，UART)接口，移动产业处理器接口(mobile industry processor interface，MIPI)，通用输入输出(general-purposeinput/output，GPIO)接口，用户标识模块(subscriber identity module，SIM)接口，和/或通用串行总线(universal serial bus，USB)接口等。可以理解的是，本发明实施例示意的各模块间的接口连接关系，只是示意性说明，并不构成对电子设备100的结构限定。在本申请另一些实施例中，电子设备100也可以采用上述实施例中不同的接口连接方式，或多种接口连接方式的组合。In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver/transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input/output (GPIO) interface, a subscriber identity module (SIM) interface, and/or a universal serial bus (USB) interface, etc. It can be understood that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation of the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.
电子设备100的无线通信功能可以通过天线1，天线2，移动通信模块150，无线通信模块160，调制解调处理器以及基带处理器等实现。The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.
天线1和天线2用于发射和接收电磁波信号。电子设备100中的每个天线可用于覆盖单个或多个通信频带。不同的天线还可以复用，以提高天线的利用率。例如：可以将天线1复用为无线局域网的分集天线。在另外一些实施例中，天线可以和调谐开关结合使用。Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.
移动通信模块150可以提供应用在电子设备100上的包括2G/3G/4G/5G等无线通信的解决方案。移动通信模块150可以包括至少一个滤波器，开关，功率放大器，低噪声放大器(low noise amplifier，LNA)等。移动通信模块150可以由天线1接收电磁波，并对接收的电磁波进行滤波，放大等处理，传送至调制解调处理器进行解调。移动通信模块150还可以对经调制解调处理器调制后的信号放大，经天线1转为电磁波辐射出去。在一些实施例中，移动通信模块150的至少部分功能模块可以被设置于处理器110中。在一些实施例中，移动通信模块150的至少部分功能模块可以与处理器110的至少部分模块被设置在同一个器件中。The mobile communication module 150 can provide solutions for wireless communications including 2G/3G/4G/5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.
调制解调处理器可以包括调制器和解调器。其中，调制器用于将待发送的低频基带信号调制成中高频信号。解调器用于将接收的电磁波信号解调为低频基带信号。随后解调器将解调得到的低频基带信号传送至基带处理器处理。低频基带信号经基带处理器处理后，被传递给应用处理器。应用处理器通过音频设备输出声音信号，或通过显示屏191显示图像或视频。在一些实施例中，调制解调处理器可以是独立的器件。在另一些实施例中，调制解调处理器可以独立于处理器110，与移动通信模块150或其他功能模块设置在同一个器件中。The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device, or displays an image or video through the display screen 191. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.
无线通信模块160可以提供应用在电子设备100上的包括无线局域网(wirelesslocal area networks，WLAN)(如无线保真(wireless fidelity，Wi-Fi)网络)，蓝牙(bluetooth，BT)，全球导航卫星系统(global navigation satellite system，GNSS)，调频(frequency modulation，FM)，近距离无线通信技术(near field communication，NFC)，红外技术(infrared，IR)等无线通信的解决方案。无线通信模块160可以是集成至少一个通信处理模块的一个或多个器件。无线通信模块160经由天线2接收电磁波，将电磁波信号调频以及滤波处理，将处理后的信号发送到处理器110。无线通信模块160还可以从处理器110接收待发送的信号，对其进行调频，放大，经天线2转为电磁波辐射出去。The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.
在一些实施例中，电子设备100的天线1和移动通信模块150耦合，天线2和无线通信模块160耦合，使得电子设备100可以通过无线通信技术与网络以及其他设备通信。所述无线通信技术可以包括全球移动通讯系统(global system for mobile communications，GSM)，通用分组无线服务(general packet radio service，GPRS)，码分多址接入(codedivision multiple access，CDMA)，宽带码分多址(wideband code division multipleaccess，WCDMA)，时分码分多址(time-division code division multiple access，TD-SCDMA)，长期演进(long term evolution，LTE)，BT，GNSS，WLAN，NFC，FM，和/或IR技术等。所述GNSS可以包括全球卫星定位系统(global positioning system，GPS)，全球导航卫星系统(global navigation satellite system，GLONASS)，北斗卫星导航系统(beidounavigation satellite system，BDS)，准天顶卫星系统(quasi-zenith satellitesystem，QZSS)和/或星基增强系统(satellite based augmentation systems，SBAS)。In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and/or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and/or a satellite based augmentation system (SBAS).
电子设备100通过GPU，显示屏191，以及应用处理器等实现显示功能。GPU为图像处理的微处理器，连接显示屏191和应用处理器。GPU用于执行数学和几何计算，用于图形渲染。处理器110可包括一个或多个GPU，其执行程序指令以生成或改变显示信息。The electronic device 100 implements the display function through a GPU, a display screen 191, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 191 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.
电子设备100可以通过ISP，摄像头190，视频编解码器，GPU，显示屏191以及应用处理器等实现拍摄功能。The electronic device 100 can realize the shooting function through the ISP, the camera 190, the video codec, the GPU, the display screen 191 and the application processor.
外部存储器120可以用于连接外部存储卡，例如Micro SD卡，实现扩展电子设备100的存储能力。外部存储卡通过外部存储器120与处理器110通信，实现数据存储功能。例如将音乐，视频等文件保存在外部存储卡中。The external memory 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory 120 to implement a data storage function. For example, files such as music and videos are stored in the external memory card.
内部存储区121可以用于存储计算机可执行程序代码，所述可执行程序代码包括指令。处理器110通过运行存储在内部存储区121的指令，从而执行电子设备100的各种功能应用以及数据处理。内部存储区121可以包括存储程序区和存储数据区。其中，存储程序区可存储操作系统，至少一个功能所需的应用程序(比如声音播放功能，图像播放功能等)等。存储数据区可存储电子设备100使用过程中所创建的数据(比如音频数据，电话本等)等。此外，内部存储区121可以包括高速随机存取存储器，还可以包括非易失性存储器，例如至少一个磁盘存储器件，闪存器件，通用闪存存储器(universal flash storage，UFS)等。The internal storage area 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal storage area 121. The internal storage area 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal storage area 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.
示例性的，内部存储区121或外部存储器120用于存储上文所描述的第一图像、第二图像、第一图像中的目标人物的历史图像(包含优质图像)。Exemplarily, the internal storage area 121 or the external memory 120 is used to store the first image, the second image, and the historical image (including the high-quality image) of the target person in the first image described above.
图13是本申请实施例的电子设备100的软件结构框图。FIG. 13 is a software structure block diagram of the electronic device 100 according to an embodiment of the present application.
分层架构将软件分成若干个层，每一层都有清晰的角色和分工。层与层之间通过软件接口通信。在一些实施例中，将系统分为四层，从上至下分别为应用层，应用框架层，运行时(Runtime)和系统库，硬件抽象层，以及内核层。The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the system is divided into four layers, from top to bottom: application layer, application framework layer, runtime and system library, hardware abstraction layer, and kernel layer.
应用层可以包括一系列应用程序包。The application layer may include a series of application packages.
如图13所示，应用层中的应用程序包可以包括但不限于相机，图库，图像增强模块，系统管家等原因程序。该应用层中还可以包括通话、音乐、视频、短信等应用，本文对此不做限定。As shown in Figure 13, the application packages in the application layer may include but are not limited to camera, gallery, image enhancement module, system manager and other programs. The application layer may also include applications such as calls, music, video, and text messages, which are not limited in this article.
其中，图像增强模块用于执行本申请提供的图像增强方法。示例性的，如图9所示，相机、图库、以及图像增强模块可以配合执行本申请提供的图像增强方法。The image enhancement module is used to execute the image enhancement method provided in the present application. Exemplarily, as shown in FIG9 , the camera, the image library, and the image enhancement module can cooperate to execute the image enhancement method provided in the present application.
在另外一些可能的实现方式中，该图像增强模块也可以位于该软件构架的其他层级中，例如应用框架层、系统库、内核层等，此处不作限定。In some other possible implementations, the image enhancement module may also be located in other layers of the software architecture, such as an application framework layer, a system library, a kernel layer, etc., which is not limited here.
应用框架层为应用层的应用程序提供应用编程接口(application programminginterface，API)和编程框架。应用框架层包括一些预先定义的函数。The application framework layer provides an application programming interface (API) and a programming framework for the application programs in the application layer. The application framework layer includes some predefined functions.
如图13所示，应用框架层包括但不限于窗口管理器，内容提供器，通知管理器，电池管理。该应用框架层还可以包括视图系统，电话管理器，资源管理器等，本文对此不做限定。As shown in Figure 13, the application framework layer includes but is not limited to a window manager, a content provider, a notification manager, and a battery manager. The application framework layer may also include a view system, a phone manager, a resource manager, etc., which are not limited in this article.
示例性的，电子设备可以基于电池管理模块获取当前电量、以及确定当前是否处于充电灭屏状态。For example, the electronic device can obtain the current power level based on the battery management module, and determine whether it is currently in the charging and screen-off state.
窗口管理器用于管理窗口程序。内容提供器用来存放和获取数据，并使这些数据可以被应用程序访问。视图系统包括可视控件，例如显示文字的控件，显示图像的控件等。视图系统可用于构建应用程序。电话管理器用于提供电子设备100的通信功能。资源管理器为应用程序提供各种资源，比如本地化字符串，图标，图像，布局文件，视频文件等等。通知管理器使应用程序可以在状态栏中显示通知信息，可以用于传达告知类型的消息，可以短暂停留后自动消失，无需用户交互。The window manager is used to manage window programs. The content provider is used to store and obtain data and make the data accessible to applications. The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. The phone manager is used to provide communication functions for the electronic device 100. The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, etc. The notification manager enables applications to display notification information in the status bar, which can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction.
运行时(Runtime)包括核心库和虚拟机。Runtime负责系统的调度和管理。The runtime includes the core library and the virtual machine. The runtime is responsible for the scheduling and management of the system.
核心库包含两部分：一部分是编程语言(例如，jave语言)需要调用的功能函数，另一部分是系统的核心库。The core library consists of two parts: one part is the function that the programming language (for example, Java language) needs to call, and the other part is the core library of the system.
应用层和应用框架层可以运行在虚拟机中。虚拟机可以将应用层和应用框架层的编程文件(例如，jave文件)执行为二进制文件。虚拟机用于执行对象生命周期的管理，堆栈管理，线程管理，安全和异常的管理，以及垃圾回收等功能。The application layer and the application framework layer can run in a virtual machine. The virtual machine can execute the programming files (e.g., java files) of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.
系统库包括但不限于多个功能模块。例如：表面管理器(surface manager)，轻型数据库(SQLLite)，二维图形引擎(例如：SGL，媒体库(Media Libraries)等。该系统库还可以包括三维图形处理库(例如：OpenGL ES))等，本文对此不做限定。The system library includes but is not limited to multiple functional modules, such as surface manager, lightweight database (SQLLite), 2D graphics engine (such as SGL, Media Libraries, etc. The system library may also include 3D graphics processing library (such as OpenGL ES), etc., which is not limited in this article.
硬件抽象层可以包括图形模块、蓝牙库模块、摄像头模块、以及硬件合成器。The hardware abstraction layer may include a graphics module, a Bluetooth library module, a camera module, and a hardware synthesizer.
示例性的，电子设备可以基于该蓝牙库模块通过蓝牙数据传输方式获取第一图像。Exemplarily, the electronic device may acquire the first image through Bluetooth data transmission based on the Bluetooth library module.
内核层是硬件和软件之间的层。内核层包含但不限于显示驱动，摄像头驱动，传感器驱动，设备驱动等。该内核层还可以音频驱动，虚拟卡驱动等，本文对此不做限定。The kernel layer is the layer between hardware and software. The kernel layer includes but is not limited to display drivers, camera drivers, sensor drivers, device drivers, etc. The kernel layer can also be audio drivers, virtual card drivers, etc., which are not limited in this article.
示例性的，电子设备可以基于上述摄像模块和摄像头驱动以拍照方式获取第一图像。Exemplarily, the electronic device may acquire the first image by photographing based on the above-mentioned camera module and camera driver.
示例性的，本申请实施例提供的增强子系统10中的功能模块、人物处理子系统30中的功能模块、以及先验知识管理子系统40，可以包含于应用层中的图像增强模块，上述人物特征检索子系统20、以及图像数据库50可以包含于系统库的SQLLite。Exemplarily, the functional modules in the enhancement subsystem 10, the functional modules in the character processing subsystem 30, and the prior knowledge management subsystem 40 provided in the embodiment of the present application can be included in the image enhancement module in the application layer, and the above-mentioned character feature retrieval subsystem 20 and the image database 50 can be included in the SQLLite of the system library.
可理解的，基于具体设计与需求，本申请实施例提供的增强子系统10中的功能模块、人物特征检索子系统20、人物处理子系统30中的功能模块、先验知识管理子系统40、以及图像数据库50还可以包含于位于其他层级，本文对此不做限定。It is understandable that, based on specific designs and requirements, the functional modules in the enhancement subsystem 10, the character feature retrieval subsystem 20, the functional modules in the character processing subsystem 30, the prior knowledge management subsystem 40, and the image database 50 provided in the embodiment of the present application may also be included in other levels, and this document does not limit this.
上述实施例中所用，根据上下文，术语“当…时”可以被解释为意思是“如果…”或“在…后”或“响应于确定…”或“响应于检测到…”。类似地，根据上下文，短语“在确定…时”或“如果检测到(所陈述的条件或事件)”可以被解释为意思是“如果确定…”或“响应于确定…”或“在检测到(所陈述的条件或事件)时”或“响应于检测到(所陈述的条件或事件)”。As used in the above embodiments, the term "when..." may be interpreted to mean "if..." or "after..." or "in response to determining..." or "in response to detecting...", depending on the context. Similarly, the phrases "upon determining..." or "if (the stated condition or event) is detected" may be interpreted to mean "if determining..." or "in response to determining..." or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)", depending on the context.
在上述实施例中，可以全部或部分地通过软件、硬件、固件或者其任意组合来实现。当使用软件实现时，可以全部或部分地以计算机程序产品的形式实现。所述计算机程序产品包括一个或多个计算机指令。在计算机上加载和执行所述计算机程序指令时，全部或部分地产生按照本申请实施例所述的流程或功能。所述计算机可以是通用计算机、专用计算机、计算机网络、或者其他可编程装置。所述计算机指令可以存储在计算机可读存储介质中，或者从一个计算机可读存储介质向另一个计算机可读存储介质传输，例如，所述计算机指令可以从一个网站站点、计算机、服务器或数据中心通过有线(例如同轴电缆、光纤、数字用户线)或无线(例如红外、无线、微波等)方式向另一个网站站点、计算机、服务器或数据中心进行传输。所述计算机可读存储介质可以是计算机能够存取的任何可用介质或者是包含一个或多个可用介质集成的服务器、数据中心等数据存储设备。所述可用介质可以是磁性介质，(例如，软盘、硬盘、磁带)、光介质(例如DVD)、或者半导体介质(例如固态硬盘)等。In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integration. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.
本领域普通技术人员可以理解实现上述实施例方法中的全部或部分流程，该流程可以由计算机程序来指令相关的硬件完成，该程序可存储于计算机可读取存储介质中，该程序在执行时，可包括如上述各方法实施例的流程。而前述的存储介质包括：ROM或随机存储记忆体RAM、磁碟或者光盘等各种可存储程序代码的介质。A person skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by a computer program to instruct the relevant hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned method embodiments. The aforementioned storage medium includes: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.
以上所述，以上实施例仅用以说明本申请的技术方案，而非对其限制；尽管参照前述实施例对本申请进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本申请各实施例技术方案的范围。As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310090530.9ACN118396857A (en) | 2023-01-20 | 2023-01-20 | Image processing method and electronic equipment |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310090530.9ACN118396857A (en) | 2023-01-20 | 2023-01-20 | Image processing method and electronic equipment |
| Publication Number | Publication Date |
|---|---|
| CN118396857Atrue CN118396857A (en) | 2024-07-26 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310090530.9APendingCN118396857A (en) | 2023-01-20 | 2023-01-20 | Image processing method and electronic equipment |
| Country | Link |
|---|---|
| CN (1) | CN118396857A (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN119399342A (en)* | 2025-01-03 | 2025-02-07 | 哈尔滨工业大学（威海） | Object Gaussian inverse rendering method based on global-local surface cooperative perception |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN119399342A (en)* | 2025-01-03 | 2025-02-07 | 哈尔滨工业大学（威海） | Object Gaussian inverse rendering method based on global-local surface cooperative perception |
| Publication | Publication Date | Title |
|---|---|---|
| US12354228B2 (en) | Generating 3D data in a messaging system | |
| US11488359B2 (en) | Providing 3D data for messages in a messaging system | |
| CN110163810B (en) | Image processing method, device and terminal | |
| CN113129312B (en) | Image processing method, device and equipment | |
| CN109961453B (en) | An image processing method, device and equipment | |
| CN109087376B (en) | Image processing method, device, storage medium and electronic device | |
| CN112262563B (en) | Image processing method and electronic device | |
| CN100456804C (en) | Facial image compensation device and method | |
| CN107249105A (en) | Exposure compensation, device and terminal device | |
| CN108810406B (en) | Portrait light effect processing method, device, terminal and computer-readable storage medium | |
| WO2020192692A1 (en) | Image processing method and related apparatus | |
| US20160140748A1 (en) | Automated animation for presentation of images | |
| CN110288534A (en) | Image processing method, device, electronic device, and storage medium | |
| CN108109161A (en) | Video data real-time processing method and device based on adaptive threshold fuzziness | |
| CN106815803A (en) | The processing method and processing device of picture | |
| CN117745620B (en) | Image processing method and electronic equipment | |
| CN112839167A (en) | Image processing method, image processing device, electronic equipment and computer readable medium | |
| US20240305878A1 (en) | Inclusive camera | |
| WO2021180046A1 (en) | Image color retention method and device | |
| WO2021128593A1 (en) | Facial image processing method, apparatus, and system | |
| CN112036209A (en) | Portrait photo processing method and terminal | |
| JP5896204B2 (en) | Image processing apparatus and program | |
| CN118396857A (en) | Image processing method and electronic equipment | |
| CN117689782B (en) | Method, device, equipment and storage medium for generating poster image | |
| CN114066724A (en) | Image processing method, intelligent terminal and storage medium |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | Country or region after:China Address after:Unit 3401, unit a, building 6, Shenye Zhongcheng, No. 8089, Hongli West Road, Donghai community, Xiangmihu street, Futian District, Shenzhen, Guangdong 518040 Applicant after:Honor Terminal Co.,Ltd. Address before:3401, unit a, building 6, Shenye Zhongcheng, No. 8089, Hongli West Road, Donghai community, Xiangmihu street, Futian District, Shenzhen, Guangdong Applicant before:Honor Device Co.,Ltd. Country or region before:China | |
| CB02 | Change of applicant information |