CN114331824B - Image processing method, device, electronic device and storage medium - Google Patents

Abstract

The embodiment of the disclosure provides an image processing method, an image processing device, electronic equipment and a storage medium. The method comprises the steps of responding to special effect triggering operation, collecting an image to be processed comprising a target object, processing the image to be processed based on a target pupil characteristic adjustment model and at least one target condition label corresponding to the image to be processed to obtain a target special effect image, and displaying the target special effect image on a target display interface. According to the technical scheme provided by the embodiment of the disclosure, the pupil is finely adjusted based on the pupil attribute condition label, the special effect is added to the eyes, the generated special effect diagram is consistent with the condition label, the accuracy of adding the special effect to the eyes is improved, and when the special effect diagram is displayed on the display interface, the displayed image is more vivid, so that the technical effect of meeting the user experience requirement is achieved.

Description

Image processing method, device, electronic equipment and storage medium

Technical Field

The embodiment of the disclosure relates to the technical field of image processing, in particular to an image processing method, an image processing device, electronic equipment and a storage medium.

Background

With the development of science and technology, more and more application software is going into the life of users, and the amateur life of users, such as short video application software, is gradually enriched. The user can record life in a video, photo or the like manner. Alternatively, the corresponding video may be photographed based on short video software. In order to further increase the richness and interest of video content, special effects are usually performed on video frames in a video.

When adding special effects to an image, the special effects are usually processed by adopting a PS (photo processing software) software and the like. The method can also adopt a pre-trained countermeasure model to carry out special effect processing on the image so as to obtain a special effect diagram.

When the special effect diagram is determined in the mode, the problems of large calculated amount and poor reality of the special effect image exist, so that the problem of poor use experience of a user is caused.

Disclosure of Invention

The embodiment of the disclosure provides an image processing method, an image processing device, electronic equipment and a storage medium, so as to realize multi-aspect adjustment of pupil characteristics, thereby improving the technical effect of the reality of a special effect image.

In a first aspect, an embodiment of the present disclosure provides an image processing method, including:

responding to the special effect triggering operation, and collecting an image to be processed comprising a target object;

Processing the image to be processed and at least one target condition label corresponding to the image to be processed based on a target pupil characteristic adjustment model to obtain a target special effect image, wherein pupil characteristics of a target object in the target special effect image are matched with the at least one target condition label;

and displaying the target special effect image on a target display interface.

In a second aspect, an embodiment of the present disclosure further provides an image processing apparatus, including:

The image acquisition module to be processed is used for responding to the special effect triggering operation and acquiring an image to be processed comprising the target object;

The target special effect image acquisition module is used for processing the image to be processed and at least one target condition label corresponding to the image to be processed based on a target pupil characteristic adjustment model to obtain a target special effect image, wherein pupil characteristics of a target object in the target special effect image are matched with the at least one target condition label;

And the image display module is used for displaying the target special effect image on a target display interface.

In a third aspect, embodiments of the present disclosure further provide an electronic device, the device including:

One or more processors;

storage means for storing one or more programs,

The one or more programs, when executed by the one or more processors, cause the one or more processors to implement the image processing method as described in any of the embodiments of the present disclosure.

In a fourth aspect, the disclosed embodiments also provide a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements an image processing method according to any of the disclosed embodiments.

According to the technical scheme, the target object-containing image to be processed is acquired by responding to the special effect triggering operation, the target object-containing image to be processed is further processed based on the target pupil characteristic adjustment model, and at least one target condition label corresponding to the target object-containing image is processed to obtain a target special effect image, and the target special effect image is displayed on the target display interface, so that the pupil characteristic is adjusted in multiple aspects, and the technical effects of the richness and the reality of the image content are improved.

Drawings

The above and other features, advantages, and aspects of embodiments of the present disclosure will become more apparent by reference to the following detailed description when taken in conjunction with the accompanying drawings. The same or similar reference numbers will be used throughout the drawings to refer to the same or like elements. It should be understood that the figures are schematic and that elements and components are not necessarily drawn to scale.

Fig. 1 is a flowchart of an image processing method according to a first embodiment of the disclosure;

fig. 2 is a flowchart of an image processing method according to a second embodiment of the disclosure;

fig. 3 is a flowchart of an image processing method according to a third embodiment of the present disclosure;

FIG. 4a is a flowchart of determining a first training sample according to a third embodiment of the present disclosure;

FIG. 4b is a schematic diagram of a first theoretical image provided by a third embodiment of the disclosure;

Fig. 5 is a flowchart of an image processing method according to a fourth embodiment of the present disclosure;

Fig. 6a is a schematic view of an eye structure segmentation image provided in accordance with a fourth embodiment of the present disclosure;

Fig. 6b is a schematic diagram of an eye reconstructed image with pupil position differences provided by a fourth embodiment of the present disclosure;

fig. 7 is a flowchart of an image processing method according to a fifth embodiment of the present disclosure;

fig. 8 is a schematic view of an eye image of pupil size variation provided by embodiment five of the present disclosure;

Fig. 9 is a flowchart of an image processing method according to a sixth embodiment of the disclosure;

fig. 10a is a schematic view of an eye reconstruction image according to a sixth embodiment of the disclosure;

fig. 10b is a schematic diagram of an image to be used provided in a sixth embodiment of the disclosure;

FIG. 10c is a schematic diagram of an image to be used according to a sixth embodiment of the disclosure;

fig. 10d is a schematic diagram of an image to be used provided in a sixth embodiment of the present disclosure;

fig. 11 is a block diagram of an image processing apparatus provided in a seventh embodiment of the present disclosure;

fig. 12 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure.

Detailed Description

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure have been shown in the accompanying drawings, it is to be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, but are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustration purposes only and are not intended to limit the scope of the present disclosure.

It should be understood that the various steps recited in the method embodiments of the present disclosure may be performed in a different order and/or performed in parallel. Furthermore, method embodiments may include additional steps and/or omit performing the illustrated steps. The scope of the present disclosure is not limited in this respect.

The term "including" and variations thereof as used herein are intended to be open-ended, i.e., including, but not limited to. The term "based on" is based at least in part on. The term "one embodiment" means "at least one embodiment," another embodiment "means" at least one additional embodiment, "and" some embodiments "means" at least some embodiments. Related definitions of other terms will be given in the description below.

It should be noted that the terms "first," "second," and the like in this disclosure are merely used to distinguish between different devices, modules, or units and are not used to define an order or interdependence of functions performed by the devices, modules, or units. It should be noted that references to "one", "a plurality" and "a plurality" in this disclosure are intended to be illustrative rather than limiting, and those of ordinary skill in the art will appreciate that "one or more" is intended to be understood as "one or more" unless the context clearly indicates otherwise.

Before the present technical solution is introduced, an application scenario may be illustrated. The technical scheme of the disclosure can be applied to any picture needing special effect display, such as in the video shooting process, special effect processing can be carried out on a shot object to obtain a displayed target special effect diagram, and the technical scheme of the disclosure can also be applied to the still image shooting process, for example, after the terminal equipment shoots an image by a camera, the shot image is processed into a special effect image to carry out special effect display. In this embodiment, the added special effects may be eye special effects of various subjects, for example, eye special effects may be special effects of pupil dilation, pupil constriction, and relative shift of the pupil center point and the eye center point. In this embodiment, the target object may be a user, or may be various pets photographed.

Example 1

Fig. 1 is a schematic flow chart of an image processing method provided in an embodiment of the present disclosure, where the embodiment of the present disclosure is applicable to a situation that an eye image of a target object is processed into a special effect image and displayed in any image display scene supported by the internet, the method may be performed by an image processing apparatus, and the apparatus may be implemented in a form of software and/or hardware, optionally, by an electronic device, and the electronic device may be a mobile terminal, a PC end, a server, or the like. Any image presentation scenario may be executed by the server, the client, or a combination of the client and the server.

As shown in fig. 1, the method of the present embodiment includes:

s110, responding to the special effect triggering operation, and collecting an image to be processed comprising the target object.

It should be noted that the foregoing has briefly described various applicable scenarios, and will not be specifically described herein. The device for executing the image processing method provided by the embodiment of the present disclosure may be integrated in application software supporting an image processing function, and the software may be installed in an electronic device, where the electronic device may be a mobile terminal or a PC terminal, and so on. The application software may be a type of software for image/video processing, and specific application software thereof is not described herein in detail, as long as image/video processing can be implemented. The method can also be a specially developed application program to realize the addition of special effects and special effect display software or be integrated in a corresponding page, and a user can realize special effect addition processing through the page integrated in the PC side.

The image to be processed is understood to be the image that needs to be processed. The image may be an image acquired based on the application software, or may be an image stored in advance by the application software from the storage space. In practical application, an image including a target object may be photographed in real time or periodically based on application software, and at this time, special effects may be directly added to the object. Or after detecting that the user triggers the special effect adding control, sending the image to the server, and adding the special effect for the target object in the acquired image to be processed by the server. Correspondingly, the image to be processed can comprise a target object, and the target main body can be a user or a pet. It should be noted that, the processing may also be performed on a video frame corresponding to the captured video, for example, a target object corresponding to the captured video may be preset, and when the target object is detected in an image corresponding to the video frame, the image corresponding to the video frame may be used as an image to be processed, so that the target object in each video frame image in the video may be processed in a special effect subsequently.

It should be noted that the number of the target objects in the same shooting scene may be one or more, and the technical scheme provided by the disclosure may be adopted to determine the special effect display diagram no matter one or more.

Specifically, when it is detected that a special effect needs to be added to a target object in an image to be processed, the image to be processed including the target object may be acquired, so as to add the special effect to the target object in the image to be processed, thereby obtaining a target special effect map corresponding to the image to be processed.

In the embodiment, the special effect triggering operation comprises at least one of detecting that a target object is included in a visual field area, detecting that the target object triggers a target special effect action and detecting that a special effect generating control is triggered.

The special effect generation control can be a key displayed on a display interface of the application software, and the triggering representation of the key needs to acquire an image to be processed and process the special effect of the image to be processed. In practical application, if the user triggers the key, it can be considered that the image function of special effect display is to be triggered, that is, a corresponding special effect needs to be added for the target object. Wherein, what the added special effect is can be consistent with the user-triggered special effect. The target special effect action may be a preset limb action, and the preset limb action may be matched with the added special effect, or it may be understood that different special effects correspond to different limb actions. The target special effect action in the technical scheme can be the action of turning the head or the action of blinking, and the intelligent of special effect identification and addition is further improved by the mode. For example, whether the limb movement of the target object in the visual field is consistent with the preset limb movement can be determined according to the shooting visual field of the mobile terminal, if so, the special effect adding operation is triggered, for example, the preset limb movement is a blink movement, and if the limb movement of the target object triggers the blink movement, the special effect triggering operation is triggered. In another implementation manner, whether the visual field range includes the target object is judged according to the shooting visual field range of the mobile terminal, if so, the special effect adding operation is triggered, for example, a pet cat can be preset as the target object, and when the pet cat is detected to be included in the visual field area, the special effect triggering operation is triggered.

S120, processing the image to be processed and at least one target condition label corresponding to the image to be processed based on a target pupil characteristic adjustment model to obtain a target special effect image.

The target pupil characteristic adjustment model is a model for adjusting pupil attributes. The target condition label may be understood as a pupil attribute label. The target condition label may be a pupil position label and/or a pupil size label. The target special effect map is an image that matches the image to be processed and the at least one target condition label. For example, if the target condition label is label 1 with a pupil radius of 5mm, a target special effect diagram with a pupil size of 5mm can be obtained after the image to be processed and the label 1 are processed based on the target pupil characteristic adjustment model.

The target condition label which is the same as the pupil characteristic of the target object can be set according to the actual requirement, and the target condition label can be set according to the final requirement effect. In practical application, in order to improve interaction efficiency with a user, a program built in a target condition label may be used to invoke a corresponding target condition label when an image to be processed is processed.

Specifically, after receiving the image to be processed, the server may input the image to be processed and at least one preset target condition label into the target pupil adjustment model, to obtain a target special effect diagram matched with the target condition label.

And S130, displaying the target special effect image on a target display interface.

Specifically, after the target special effect image is obtained, the target special effect image may be displayed on the target display interface in order to enable the user to understand the final special effect image. The target display interface is a display interface of the corresponding terminal equipment when the image to be processed is uploaded.

According to the technical scheme, the target object-containing image to be processed is acquired by responding to the special effect triggering operation, the target object-containing image to be processed is further processed based on the target pupil characteristic adjustment model, and at least one target condition label corresponding to the target object-containing image is processed to obtain a target special effect image, and the target special effect image is displayed on the target display interface, so that the pupil characteristic is adjusted in multiple aspects, and the technical effects of the richness and the reality of the image content are improved.

Example two

Fig. 2 is a flow chart of an image processing method according to a second embodiment of the disclosure, and S120 is further refined based on the foregoing embodiment. Wherein, the technical terms identical to or corresponding to the above embodiments are not repeated herein.

As shown in fig. 2, the method specifically includes the following steps:

S210, responding to the special effect triggering operation, and collecting an image to be processed comprising the target object.

S220, determining at least one target condition label corresponding to the image to be processed.

In the embodiment, determining at least one target condition label comprises obtaining at least one preset target condition label or determining at least one target condition label corresponding to a target object according to the gesture information of the target object in the image to be processed.

It can be understood that the target condition label can be pre-stored in the database, and when what special effect needs to be generated, the target condition label matched with the special effect can be directly called. After receiving the image to be processed, the target condition label and the image to be processed can be used as input of a target pupil adjustment model so as to obtain a target special effect diagram.

In order to improve the adaptation degree of the target object in the image to be processed and the added special effect, a condition label which is suitable for the form information can be determined according to the form information corresponding to the target object in the image to be processed and used as the target condition label. If the target object is tilted and eyes are not completely exposed, the eye state of the target object can be analyzed through an algorithm to determine what size the eyes are processed to have the best special effect, and if the pupil size of the best effect is 5mm, the pupil size of 5mm can be used as a target condition label. The target condition label can be determined under the condition that the adaptation degree of the target object and the added special effect is improved and the special effect requirement of the user is met, for example, the special effect requirement can be the added special effect, the happy special effect, the vital energy special effect and the like, and the target condition label under the condition is determined.

The gesture information may be a head deflection angle, or an eye state, or a facial expression.

It can be understood that the target condition label is determined according to the gesture information of the target object in the image to be processed, for example, the gesture information of the target object can be analyzed by using an algorithm, and then, according to the requirement that the specific effect is actually required to be made, it can be determined which condition label is added to the target object by the gesture information, and the eye specific effect is the best effect. For example, when a cat is to be given a question of expression, an algorithm may be used to analyze what size the cat eye is adjusted to, and what position the pupil is adjusted to is the best, and the pupil size and the pupil position at that time may be used as target condition labels.

Before determining the target condition label, the condition labels corresponding to different positions of the pupil in the eye can be determined by dividing and calibrating the positions of the eye. Optionally, the label corresponding to the pupil center point and the eye center point when the pupil center point is overlapped is recorded as 0.5, the label can be sequentially reduced if the pupil center point is left-deviated relative to the eye center point, and the label can be sequentially increased if the pupil center point is right-shifted relative to the eye center point. The leftmost tag 0 and the rightmost tag 1. Pupil size labels may be defined in terms of the radius of the pupil, alternatively, the pupil radius may be 2mm, 5mm, 8mm, pupil size labels may be 2,5, 8, etc. The target condition label may be determined from a plurality of pupil size labels and pupil position labels that are set in advance.

In this embodiment, the at least one target condition label includes a target size label corresponding to a target display size of the pupil, and/or a target position label corresponding to a target relative display position of the pupil in the eye.

It is understood that the target size label is used to characterize the pupil display radius. A target location tag for a tag that characterizes information about the deviation of the pupil center in the eye. The target size tag and the target position tag may be regarded as target condition tags. In this embodiment, the number of the labels in the target condition label and the target pupil adjustment model obtained by training have a certain association, and the specific manner of the method may be that if the target pupil feature adjustment model is obtained by training under the conditions of fixed position label and variable size label, at least one target condition label includes a target size label, if the target pupil feature adjustment model is obtained by training under the conditions of fixed size label and variable size label, at least one target condition label includes a target position label, and if the target pupil feature adjustment model is obtained by training under the conditions of variable size label and variable size label, at least one target condition label includes a target position label and a target size label.

It will be appreciated that the target pupil feature adjustment model may be pre-trained using condition labels, e.g., the model may be trained with the condition labels unchanged, the condition label partially changed, or a combination of condition labels changed.

For example, if the transformed condition label is a position label, the size label may be fixed. And training a p2p neural network through different position labels and corresponding images to be trained, so as to obtain a target pupil characteristic adjustment model through training. The pupil characteristic adjustment model can be input into a position label and a corresponding image to be processed, the pupil position in the obtained target special effect diagram is consistent with the position label, and the pupil size is consistent with the fixed size label. If the transformed condition label is a size label, the position label may be fixed. And training a p2p neural network through different size labels and corresponding images to be trained, so as to obtain a target pupil characteristic adjustment model through training. The pupil characteristic adjustment model can be input into a size label and a corresponding image to be processed, the pupil size in the obtained target special effect diagram is consistent with the size label, and the pupil position is consistent with the fixed position label. If the target pupil characteristic adjustment model is obtained by training under the conditions of fixed position labels and variable size labels, the position labels and the size labels can be randomly combined, the combined labels and the original image are used as the input of the model to be trained, and the corresponding theoretical image is used as the model output so as to train and obtain the target pupil characteristic adjustment model.

S230, taking the at least one target condition label and the image to be processed as input of the target pupil adjustment model, and obtaining the target special effect diagram.

In this embodiment, the image to be processed and the selected at least one target condition label may be input to the target pupil adjustment model. The model can process the image to be processed and output a target special effect diagram under a target condition label, for example, if the target condition label is pupil size 8mm and pupil position 0.5, a special effect diagram with pupil radius changed to 8mm and pupil moving to the middle position of the eye can be generated.

In the embodiment, the specific mode of processing the image to be processed based on the target pupil feature adjustment model may be that if at least one target condition label includes a target size label and a target position label, the pupil of the target object in the image to be processed is adjusted to match the target size label and the target position label based on the target pupil feature adjustment model to obtain a target special effect image, if at least one target condition label includes the target size label, the pupil of the target object in the image to be processed is adjusted to match the target size label and the position label used when the target pupil feature adjustment model is obtained by training based on the target pupil feature adjustment model, and if at least one target condition label includes the target position label, the pupil of the target object in the image to be processed is adjusted to match the target position label and the size label used when the target pupil feature adjustment model is obtained by training based on the target pupil feature adjustment model.

S240, displaying the target special effect image on a target display interface.

According to the technical scheme, the target condition label which corresponds to the image to be processed and comprises the target size label and the target position label is obtained, and then at least one target condition label and the image to be processed are used as the input of the target pupil adjustment model, so that the target special effect diagram is obtained, the pupil is finely adjusted based on the pupil size and the position condition label, the accuracy and the richness of the eye special effect addition are improved, and the technical effect of user experience is further improved.

Example III

Fig. 3 is a schematic flow chart of an image processing method provided in a third embodiment of the present disclosure, on the basis of the foregoing embodiment, a target pupil feature adjustment model may be trained in advance, so as to determine a target special effect diagram based on the target pupil feature adjustment model, and a specific implementation manner may refer to a technical scheme of the present embodiment. Wherein, the technical terms identical to or corresponding to the above embodiments are not repeated herein.

As shown in fig. 3, the method specifically includes the following steps:

S310, a first training sample set is acquired.

In order to improve the accuracy of the model, training samples can be acquired as much and as much as possible. The first training sample set includes a plurality of first training samples including a first raw input image and a first theoretical image. The first original input image may be an image captured by the imaging device or an image reconstructed based on an image reconstruction model, or may be an image stored in advance in the storage space. At this time, no corresponding special effects are added to the first original input image, i.e., pupil features in the unmodified image. Meanwhile, the first original input image must include a target portion of the target object. The target site may be an eye site. The first theoretical image is generated based on a target pupil position adjustment model and a target pupil size adjustment model obtained through pre-training. At this time, the first theoretical image is an image that is expected to be output by the pupil characteristic adjustment model to be trained.

The target pupil position adjustment model is used for adjusting the pupil position of the object in the first original input image. And the target pupil size adjustment model is used for adjusting the pupil size of the object in the first original input image. The original input image may be processed based on the target pupil position adjustment model and the target pupil size adjustment model in sequence to obtain a first theoretical image of the training target pupil feature adjustment model.

In this embodiment, determining the first theoretical image of each training sample may be to obtain a first original input image, input the first original input image and the pupil position label to be displayed into the target pupil position adjustment model to obtain an image to be used, and input the image to be used and the pupil size label to be displayed into the target pupil size adjustment model to obtain the first theoretical image.

The pupil position label to be displayed refers to the label of the position information of the pupil in the eye after the expected target pupil position model processes the image. For example, if the pupil is expected to be shifted by twenty percent from the eye center point, the pupil position label to be displayed may be 0.2, and if the pupil is expected to be shifted by ninety percent from the eye center point, the pupil position label to be displayed may be 0.9. The pupil position label 0.9 to be displayed and the first original input image can be input into the target pupil position adjustment model at the same time, and a to-be-used image with the pupil offset of ninety percent of the eye center point is obtained. The pupil size label to be displayed is a label of the pupil size after the image is processed by the desired target pupil size model. If the radius of the pupil in the eye is expected to be 5mm, the pupil size label to be displayed may be 5, and if the radius of the pupil in the eye is expected to be 2mm, the pupil size label to be displayed is 2. The pupil size label 5 to be displayed and the ninety percent of the pupil offset center point image can be used as the input of a target pupil size adjustment model to obtain the ninety percent of the pupil offset center point, and meanwhile, the pupil size is 5mm of the first theoretical image.

By adopting the mode, the first theoretical images corresponding to the first original input images can be obtained, so that the first training sample is determined based on the first original input images and the corresponding first theoretical images. A first set of training samples is determined based on each first training sample.

The process of determining the first training sample may be that, as shown in fig. 4a, the first original input image (e.g. a in fig. 4 b) and the pupil position label to be displayed are input into the target pupil position adjustment model, so as to obtain an image to be used in which the pupil moves to the specified position. The image to be used and the pupil size label to be displayed are input into a target pupil size adjustment model, a special effect diagram of pupil size change to a fixed size is obtained, and the obtained special effect diagram is a first theoretical image of pupil position movement and size change, see B in fig. 4B. Is merely a schematic diagram and is not limiting.

In practical application, the first original input image may be processed by the target pupil size adjustment model and then processed by the pupil size model, so as to obtain the first theoretical image. The specific processing order of the model is not limited as long as the first theoretical image can be obtained. On the basis of the above solution, there may be a target pupil adjustment model for which a pupil size or pupil position is desired to be fixed, in which case the optimization may be performed in determining the first theoretical image.

Optionally, if the pupil size label to be displayed is a fixed value, the first theoretical images are images with the same pupil size and different pupil positions, if the pupil position label to be displayed is a fixed value, the first theoretical images are images with the same pupil position and different pupil sizes, and if the pupil size label to be displayed changes, the pupil position label to be displayed changes, the first theoretical images are images with the different pupil positions and different pupil sizes.

It will be appreciated that if a first theoretical image with a fixed pupil size is desired, the pupil size label to be displayed may be kept at a fixed value after each image to be used is obtained. And taking the image to be used and the fixed pupil size label to be displayed together as the input of a target pupil size adjustment model to obtain a first theoretical image with fixed pupil size and certain deviation of pupil position. At this time, the pupil sizes of all the first theoretical images are the same. If a first theoretical image is desired that is constant if the pupil position is desired, the pupil size label to be displayed may be set to a fixed value. And inputting the first original input image and the fixed pupil position label to be displayed into the target pupil position adjustment model, and obtaining the images to be used with the same pupil positions. Further, the image to be used and the size label to be displayed which can be changed are input into a target pupil size adjustment model, and a first theoretical image with unchanged pupil position but changed pupil size is obtained. If a first theoretical image with fixed pupil size and pupil position is desired, it may be stated that the pupil size label to be displayed and the pupil position label to be displayed are both fixed values. If a first theoretical image is desired where pupil size and pupil position are varying, the first theoretical image may be determined in the manner set forth in detail above.

S320, aiming at each first training sample, taking a first original image of the current first training sample as input of a pupil characteristic adjustment model to be trained, taking a first theoretical image as output of the pupil characteristic adjustment model to be trained, and adjusting model parameters of the pupil characteristic adjustment model to be trained.

The model parameters in the pupil characteristic adjustment model to be trained are default values. Correcting model parameters in the pupil characteristic adjustment model to be trained through the training sample to obtain the target pupil characteristic adjustment model.

The processing method is the same for each training sample, and one of the training samples is taken as an example for processing.

Specifically, a first original input image in a current training sample is input to a pupil characteristic adjustment model to be trained, a first theoretical image is used as output of the pupil characteristic adjustment model to be trained, so that model parameters in the pupil characteristic adjustment model to be trained are adjusted, and the model parameters are continuously adjusted, so that the pupil characteristic adjustment model to be trained can output expected images when the original image is input.

S330, converging a loss function of the pupil characteristic adjustment model to be trained as a training target to obtain a target pupil characteristic adjustment model.

The method comprises the steps of taking convergence of a preset loss function as a training target, and taking the model as a target pupil characteristic adjustment model when judging that the preset loss function of the pupil characteristic adjustment model to be trained is converged.

It should be noted that, because the model parameters in the pupil feature adjustment model to be trained are not corrected, the specific image output by the model may have corresponding differences with the first theoretical image corresponding to the current training sample, and the model parameters in the pupil feature adjustment model to be trained may be continuously corrected, so that the pupil feature adjustment model to be trained may output the desired first theoretical image.

S340, responding to the special effect triggering operation, and collecting the image to be processed comprising the target object.

S350, processing the image to be processed and at least one target condition label corresponding to the image to be processed based on the target pupil characteristic adjustment model to obtain a target special effect image.

S360, displaying the target special effect image on a target display interface.

According to the technical scheme, a first theoretical image in a training sample is determined based on a target pupil position adjustment model and a target pupil size adjustment model. The pupil characteristic adjustment model to be trained is trained based on the first original input image and the corresponding first theoretical image, and the target pupil characteristic adjustment model is obtained, so that the target pupil characteristic adjustment model adjusts pupil characteristics of the acquired image, and the technical effects of special effect adding efficiency and richness are improved.

Example IV

Fig. 5 is a schematic flow chart of an image processing method provided in a fourth embodiment of the present disclosure, on the basis of the foregoing embodiment, a target pupil position adjustment model may be trained to generate an image of pupil position change based on the target pupil position adjustment model, and further generate a corresponding first theoretical image based on the image, and a specific implementation manner may refer to a technical scheme of the present embodiment. Wherein, the technical terms identical to or corresponding to the above embodiments are not repeated herein.

As shown in fig. 5, the method specifically includes the following steps:

s410, acquiring a second training sample set.

In order to improve the accuracy of the model, training samples can be acquired as much and as much as possible. The second training sample set comprises a plurality of second training samples, and the second training samples comprise second input images with different pupil positions, second theoretical output images and pupil position labels corresponding to the second theoretical output images. The second input image and the second theoretical output image are determined based on a target pupil reconstruction model obtained through pre-training, and the target pupil reconstruction model is used for reconstructing an image consistent with the marked pupil.

At this time, the second input image is an input image of the pupil position adjustment model to be trained, and the image is a real eye image reconstructed by the target pupil model. The second theoretical output image is the image that the pupil position adjustment model to be trained is expected to output. Optionally, for the same image including the eye portion, two images of different positions of the pupil in the eye are generated based on the pupil reconstruction model. The user can mark the white region and the pupil region of the eye part in the image according to the actual requirement, reconstruct the image based on the target pupil reconstruction model, and obtain the image with the pupil having offset. Because the pupil area can be marked according to actual requirements during marking, the pupil in the reconstructed image can have a certain difference from the pupil in the original image.

For example, referring to fig. 6a, a number of images containing eyes may be acquired using an imaging device or generated based on an image generation model. The pupil area and the white area of the eye may be marked, optionally, the pupil position is randomly selected in the eye and filled with black, and the part of the orbit except for the black area is filled with white, so as to obtain the white area, and obtain a in 6a in the segmentation map. Image a is input to the target pupil reconstruction model, reconstructing B as in fig. 6 a. It should be noted that the above mentioned image is only a schematic view, and is not limited thereto.

Based on the mode, a training sample for training the pupil position adjustment model to be trained can be constructed based on the target pupil reconstruction model. Referring to fig. 6B, for the same image, two images to be input with different pupil positions are marked respectively, and the two images to be input are processed based on the target pupil reconstruction model respectively to obtain two theoretical images with different pupil positions, namely, two corresponding real eye images (as shown in a and B in fig. 6B) can be obtained. It should be noted that the above mentioned image is only a schematic view, and is not limited thereto. And (3) obtaining the sample, and outputting the pupil position adjustment model to be trained in B in 6B if the A in 6B is required to be used as the input of the pupil position adjustment model to be trained. Then the pupil position label to be displayed corresponding to the pupil position can be determined when B in 6B is obtained. Based on the above manner, the input image, the theoretical image, and the pupil position label corresponding to the theoretical image in the second training sample can be obtained.

S420, aiming at each second training sample, taking a second input image and pupil position labels of the current training sample as input of a pupil position adjustment model to be trained, taking a second theoretical output image as output of the pupil position adjustment model to be trained, and adjusting model parameters of the pupil position adjustment model to be trained.

The model parameters in the pupil position adjustment model to be trained are default values. The model may be a p2p network model.

Illustratively, with continued reference to fig. 6B, the pupil location in the B image of 6B is determined to obtain a pupil location tag. And inputting the pupil position label and the image A in the 6B into a pupil position adjustment model to be trained, taking the B in the 6B as the output of the model, and adjusting model parameters in the model.

Specifically, a second input image in the current training sample is input to the pupil position adjustment model to be trained, the second theoretical output image is used as the output of the pupil position adjustment model to be trained, so that model parameters in the pupil characteristic adjustment model to be trained are adjusted, and the model parameters are continuously adjusted, so that when the original image and the position label are input, the pupil position adjustment model to be trained can output an expected image consistent with the position label.

S430, converging a loss function of the pupil position adjustment model to be trained as a training target to obtain a target pupil position adjustment model.

The method comprises the steps of taking convergence of a preset loss function as a training target, and taking the model as a target pupil position adjustment model when judging that the preset loss function of the pupil position adjustment model to be trained is converged.

It should be noted that, because the model parameters in the pupil feature adjustment model to be trained are not corrected, the specific image output by the model may have corresponding differences with the second theoretical output image corresponding to the current training sample, and the model parameters in the pupil position adjustment model to be trained may be continuously corrected, so that the pupil feature adjustment model to be trained may output the expected second theoretical output image. In this embodiment, the advantage of determining the target pupil position adjustment model is that the target pupil position adjustment model can be obtained based on the obtained first theoretical image, and further based on the first theoretical image and the corresponding first original input image.

S440, responding to the special effect triggering operation, and acquiring a to-be-processed image comprising the target object.

S450, processing the image to be processed and at least one target condition label corresponding to the image to be processed based on the target pupil characteristic adjustment model to obtain a target special effect image.

S460, displaying the target special effect image on a target display interface.

According to the technical scheme, the target pupil position adjustment model can be obtained through training by determining the second input image and the second theoretical output image, and further the theoretical image consistent with part of features in the second theoretical image can be obtained based on the target pupil position adjustment model, so that the first training sample is obtained based on the theoretical image, and the effect of determining the first training sample, accuracy and convenience are improved.

Example five

Fig. 7 is a schematic flow chart of an image processing method provided in a fifth embodiment of the present disclosure, on the basis of the foregoing embodiment, a target pupil size adjustment model may be trained to generate an image of pupil size change based on the target pupil size adjustment model, and further generate a corresponding first theoretical image based on the image, and a specific implementation manner may refer to a technical solution of the present embodiment. Wherein, the technical terms identical to or corresponding to the above embodiments are not repeated herein.

As shown in fig. 7, the method specifically includes the following steps:

s510, acquiring a third training sample set.

The third training sample set comprises a plurality of third training samples, and the third training samples comprise a third input image, a third theoretical output image and pupil size labels corresponding to the third theoretical images, wherein the third input image and the third theoretical output image are different in pupil size. The third input image refers to the input of the model. The third theoretical output image refers to the output of the model. The third input image and the third theoretical output image are determined based on a target pupil reconstruction model obtained through pre-training, and the target pupil reconstruction model is used for reconstructing an image consistent with the marked pupil.

Illustratively, a person generates a plurality of images including eyes based on an image generation model. The pupil area and the white area of the eye can be marked, and then the marked image is input into a target pupil reconstruction model to obtain a size-adjusted image. For example, referring to the original image a in fig. 8, pupil size marking is performed, the marked pupil sizes are different, and the marked image is input into a target pupil reconstruction model to obtain an eye image with pupil size change. One of the images can be used as the input of the pupil size adjustment model to be trained, and the other image can be used as the output of the pupil size adjustment model to be trained, and at this time, the input also comprises the pupil size label of the output image. In this way a plurality of third training samples may be obtained. The set of each third training sample is a third training sample set.

S520, aiming at each third training sample, taking a third input image and pupil size labels of the current training sample as input of a pupil position adjustment model to be trained, taking a third theoretical output image as output of the pupil position adjustment model to be trained, and adjusting model parameters of the pupil position adjustment model to be trained.

The model parameters in the pupil position adjustment model to be trained are default parameters.

Specifically, the third input image and the pupil size label in the current training sample may be input to the pupil size adjustment model to be trained, and the third theoretical output image is used as the output of the pupil size adjustment model to be trained. That is, the model parameters in the pupil size adjustment model to be trained can be adjusted by training the model under the condition that the input parameters and the output parameters of the pupil size adjustment model to be trained are fixed, and the model parameters are continuously adjusted, so that when the original image and the pupil size label are input, the model can output a desired third theoretical output image, and the pupil size in the third theoretical output image is matched with the input pupil size label.

S530, converging a loss function of the pupil size adjustment model to be trained as a training target, and obtaining the target pupil size adjustment model.

The convergence of the preset loss function can be used as a training target, and when the convergence of the preset loss function of the pupil size adjustment model to be trained is judged, the adjustment result accords with the scheme requirement, and the trained model is obtained, so that the target pupil size adjustment model is obtained.

It should be noted that, because the model parameters in the pupil size adjustment model to be trained are not corrected, the specific image output by the model may have corresponding differences with the third theoretical output image corresponding to the current training sample, and the model parameters in the pupil size adjustment model to be trained may be continuously corrected, so that the pupil size adjustment model to be trained may output the expected third theoretical output image.

Specifically, when the convergence of the loss function of the pupil size adjustment model to be trained is detected, the completion of the training of the pupil size adjustment model to be trained is indicated, and at this time, the iterative training can be stopped. If the loss function is detected not to be converged at present, a training sample can be further obtained to continue training the pupil size adjustment model to be trained until the loss function is converged, the pupil size adjustment model to be trained can be considered to be trained, so that the pupil size adjustment model to be trained is input with the image to be processed and the target size label, and the model can adjust the target object pupil in the image to be processed to the size corresponding to the target size label, so that the special effect image with the size change can be generated. The trained pupil size adjustment model to be trained can be used as a target pupil size adjustment model.

S540, responding to the special effect triggering operation, and collecting the image to be processed comprising the target object.

S550, processing the image to be processed and at least one target condition label corresponding to the image to be processed based on a target pupil characteristic adjustment model to obtain a target special effect image.

S560, displaying the target special effect image on a target display interface.

According to the technical scheme, the image consistent with the marked pupil is obtained based on the target pupil reconstruction model, the third input image and the third theoretical output image are correspondingly obtained to serve as training samples, the model is trained, and the trained target pupil size adjustment model is obtained, so that the pupil size change of the target object in the image is controlled based on the target pupil size adjustment model, and the efficiency and accuracy of adding special effects are improved.

Example six

Fig. 9 is a schematic flow chart of an image processing method provided in a sixth embodiment of the present disclosure, on the basis of the foregoing embodiment, a pupil reconstruction model to be trained may be trained based on a training sample, so as to obtain a target pupil reconstruction model, and a specific implementation manner of the method may refer to a technical scheme of the embodiment. Wherein, the technical terms identical to or corresponding to the above embodiments are not repeated herein.

As shown in fig. 9, the method specifically includes the following steps:

S610, determining a plurality of fourth training samples, wherein the fourth training samples comprise preprocessed images, and determining images to be used, comprising pupil marks, in the preprocessed images according to a preset image processing algorithm.

It should be noted that, before training to obtain the target pupil reconstruction model, a training sample needs to be obtained first to train the model based on the training sample. In order to improve the accuracy of the model, training samples can be acquired as much and as much as possible.

The preset image processing algorithm is used for performing pupil marking on the acquired image. The preprocessed image may be an image that contains the target object. The preprocessed image may be obtained in various ways, for example, may be an image captured by an imaging device, an image stored in advance in a storage space, or an image generated based on an image generation model.

After the preprocessed image is obtained, a theoretical output image corresponding to the preprocessed image can be determined, so that a target pupil reconstruction model is obtained through training based on the preprocessed image and the corresponding theoretical output image. The image reconstructed based on the pupil markers may be used as the image to be used. That is, a corresponding image may be constructed based on the pupil marker, and this image may be used as the image to be used. The image to be used is a gray scale image of the pupil marker. It will be appreciated that the training sample includes the pupil marker image and the reconstructed image

In the embodiment, each fourth training sample is determined by acquiring a preprocessed image comprising a target position, determining a gray level image of the target position, determining a first area and a second area in the gray level image according to a preset image processing algorithm, wherein the target position is an eye position, the first area is a pupil area, the second area is an eye white area, and obtaining an image to be used comprising pupil marks based on the first area and the second area.

The image capturing device may capture an image of the eye portion of the target object, and a plurality of images including the eyes may be acquired as the preprocessed image. To improve the accuracy of the division of the eye, the pupil area is determined by converting the pre-processed image into a gray scale map using an algorithm, the gray scale value may comprise any of values 0 to 255, where 255 represents full white and 0 represents full black. Furthermore, a pupil area and an eye white area in the eye part can be determined by using a preset image processing algorithm and corresponding gray values, and the eye white area and the pupil area are marked to obtain an input image input to the pupil reconstruction model to be trained. For example, referring to fig. 10a, after an image is acquired, a pre-processed image is obtained, and the pupil area and the white area in the eye may be marked to obtain an image to be used, such as a in fig. 10 a. And (3) performing image reconstruction processing on the A in the 10a to obtain the B in the 10 a. Is merely a schematic diagram and is not limiting.

In this embodiment, the determining the preset image processing algorithm may include a gray-scale average algorithm and a gray-scale threshold algorithm. Accordingly, the method for determining the image to be used includes at least three methods, which may specifically be:

The first mode of determining the image to be used can be that the preset image processing algorithm comprises a gray average value algorithm, a first area and a second area in a gray image are determined according to the preset image processing algorithm, the first area and the second area are determined according to the preset image processing algorithm, the gray average value of the gray image is determined, pixels lower than the gray average value are used as first type pixels, pixels higher than the gray average value are used as second type pixels, and the first area and the second area are determined based on the first type pixels and the second type pixels.

Specifically, after the preprocessed image is converted into the gray scale image, a gray scale average value of the target part in the gray scale image can be calculated. The pixel point with the gray value lower than the gray average value of the pixel point in the target part can be used as the first type pixel point, and the pixel point with the gray value higher than the gray average value can be used as the second type pixel point. The region formed by the first type pixel points may be referred to as a first region, and the region formed by the second type pixel points may be referred to as a second region. The first region may be a pupil region and the second region may be an eye white region.

The second mode of determining the image to be used may be that the preset image processing algorithm includes a gray threshold algorithm, and determining a first area and a second area in the gray image according to the preset image processing algorithm includes using pixels with gray values smaller than a preset gray threshold value as a first type of pixels and pixels with gray values greater than or equal to the preset gray threshold value as a second type of pixels, and determining the first area and the second area based on the first type of pixels and the second type of pixels.

Specifically, a preset gray value threshold is determined before the gray map of the eye region is obtained. The pixels with gray values smaller than the preset gray value threshold in the gray map can be used as the first type of pixels, and the pixels with gray values larger than or equal to the preset gray value threshold can be used as the second type of pixels. The white region and the pupil region may be determined based on the first type of pixel points and the second type of pixel points.

The third mode of determining the image to be used can be that a first pixel point to be determined, the gray average value of which is larger than a preset gray value threshold value, a second pixel point to be determined, the gray value of which is smaller than the preset gray value threshold value, and based on the first pixel point to be determined and the second pixel point to be determined, a first divided image is determined, a third pixel point to be determined, the gray average value of which is larger than the gray value average value of a target part, a fourth pixel point to be determined, the gray value of which is smaller than the gray value average value, and based on the third pixel point to be determined and the fourth pixel point to be determined, a second divided image is determined, and a pixel point intersection and a pixel point union are determined according to the first pixel point to be determined and the third pixel point to be determined in the first divided image and the second divided image, and if the ratio of the pixel point intersection and the pixel point union is larger than a preset ratio threshold value, a first area and a second area are determined based on the pixel point intersection.

In order to further improve the accuracy of the image to be used, the first pixel to be determined and the second pixel to be determined may be determined based on the first manner. And determining a third pixel point to be determined and a fourth pixel point to be determined based on the second mode. The first pixel point to be determined and the third pixel point to be determined correspond to pupil areas, and the second pixel point to be determined and the fourth pixel point to be determined correspond to eye white areas. The first segmented image may be the pupil area determined in the first manner. The second segmented image may be the pupil area determined in the second manner. The number of pixel intersections and the number of union sets in the first segmented image and the second segmented image may be determined for the same eye. A ratio threshold is determined based on the number of intersections and the number of union sets. If the ratio threshold is greater than the preset ratio threshold, the intersection area can be used as a pupil area, and the eye area outside the intersection is used as an eye white area. That is, the first region and the second region may be determined.

For example, referring to fig. 10b, an area formed by pixels having a gray value lower than the gray average value may be selected as a pupil area, and an area outside the primary pupil area in the eye portion is an eye white area. B in 10B may be represented as a gray scale map of the cat eye image, and a in 10B may be represented as an image to be used dividing the gray scale map into a pupil area and an eye white area. Referring to fig. 10c, an area formed by pixels below a preset gray value threshold may be selected as a pupil area, an area formed by pixels greater than the preset gray value threshold may be selected as an eye white area, B in 10c may be represented as a gray map of the cat eye image, and a in 10c may be represented as an image to be used after dividing the gray map into the pupil area and the eye white area. Is merely a schematic diagram and is not limiting.

Further, in order to ensure the quality of the training data pair, the IOUs (Intersection-over-Union) of the two eyes in the image can be calculated according to the first segmented image and the second segmented image respectively, so as to obtain a high-quality image to be used. The pixel point intersection and the pixel point union of the first pixel point to be determined and the third pixel point to be determined can be calculated by using an algorithm, then the ratio of the pixel point intersection to the pixel point union can be calculated by using the algorithm, the ratio can be compared with a preset ratio threshold, if the ratio is greater than the preset ratio threshold, the pixel point intersection result can be used as an image to be used, for example, see fig. 10d, wherein a is a first segmentation image, B is a second segmentation image, and C is a pixel point intersection result image. It should be noted that the above mentioned image is only a schematic view, and is not limited thereto. And taking the image to be used and the corresponding preprocessed image as a fourth training sample. Training the pupil reconstruction model to be trained based on the fourth training sample, so that the model can reconstruct a real eye image through the binary mask of the eyes.

S620, regarding each fourth training sample, taking the image to be used as an input parameter of a pupil reconstruction model to be trained, taking the preprocessed image as an output parameter of the pupil reconstruction model to be trained, and adjusting model parameters of the pupil reconstruction model to be trained.

The model parameters in the pupil reconstruction model to be trained are default parameters.

Specifically, after the fourth training sample set is obtained, the image to be used in the current training sample may be input to the pupil reconstruction model to be trained, and the corresponding preprocessed image is used as the output of the pupil reconstruction model to be trained. That is, the model parameters in the pupil reconstruction model to be trained may be adjusted in the case where both the input parameters and the output parameters of the pupil reconstruction model to be trained are fixed, so that the model may output a desired preprocessed image, which is an original image of the image to be used, when the image to be used is input.

S630, converging the loss function of the pupil reconstruction model to be trained as a training target to obtain the target pupil reconstruction model.

The convergence of the preset loss function can be used as a training target, and when the convergence of the preset loss function of the pupil reconstruction model to be trained is judged, the adjustment result accords with the scheme requirement, and the trained model is obtained, so that the target pupil reconstruction model is obtained.

It should be noted that, because the model parameters in the pupil reconstruction model to be trained are uncorrected, the reconstructed image output by the model may have corresponding differences with the preprocessed image corresponding to the current training sample, so that the model parameters in the pupil reconstruction model to be trained may be continuously corrected, so that the pupil reconstruction model to be trained may output the expected preprocessed image.

Specifically, when the convergence of the loss function of the pupil reconstruction model to be trained is detected, the completion of the training of the pupil reconstruction model to be trained is indicated, and at this time, the iterative training can be stopped. If the loss function is detected not to be converged at present, training the pupil reconstruction model to be trained by further acquiring a training sample until the loss function is converged, and considering that the pupil reconstruction model to be trained is trained, so that the model can reconstruct a real eye image, namely an unprocessed original image, according to the image to be used when the image to be used is input into the trained pupil reconstruction model to be trained. The trained pupil reconstruction model to be trained can be used as a target pupil reconstruction model.

In this embodiment, after the target pupil reconstruction model is obtained by training, training samples for training the target pupil position adjustment model and the target pupil size adjustment model may be constructed based on the target pupil reconstruction model. Further, based on the target pupil position adjustment model and the target pupil size adjustment model, a training sample for training the target pupil feature adjustment model is determined, and then the target feature pupil adjustment model is obtained through training.

S640, responding to the special effect triggering operation, and collecting the image to be processed comprising the target object.

S650, processing the image to be processed and at least one target condition label corresponding to the image to be processed based on a target pupil characteristic adjustment model to obtain a target special effect image.

And S660, displaying the target special effect image on a target display interface.

According to the technical scheme, the image to be used, which comprises the pupil mark, in the image to be processed is determined through the preset image processing algorithm, the image to be used is further used as the input parameter of the pupil reconstruction model to be trained, the image to be processed is used as the output parameter of the pupil reconstruction model to be trained, the model parameters of the pupil reconstruction model to be trained are adjusted, the trained target pupil reconstruction model is obtained, pupil mark processing on the image to be processed based on the preset image processing algorithm is achieved, the image to be used is obtained, the quality of training data is improved, and the training efficiency and accuracy of the model are improved.

Example seven

Fig. 11 is a block diagram of an image processing apparatus according to a seventh embodiment of the present disclosure, which is capable of executing the image processing method according to any of the embodiments of the present disclosure, and has functional modules and beneficial effects corresponding to the execution method. As shown in fig. 11, the apparatus specifically includes a to-be-processed image acquisition module 710, a target special effect image acquisition module 720, and an image display module 730.

The system comprises a target special effect image acquisition module 710, a target special effect image acquisition module 720 and an image display module 730, wherein the target special effect image acquisition module 710 is used for responding to a special effect triggering operation and acquiring a target object to be processed image, the target special effect image acquisition module 720 is used for processing the target image to be processed and at least one target condition label corresponding to the target image to obtain a target special effect image based on a target pupil characteristic adjustment model, pupil characteristics of the target object in the target special effect image are matched with the at least one target condition label, and the image display module 730 is used for displaying the target special effect image on a target display interface.

On the basis of the above technical solutions, the image acquisition module 710 to be processed includes a special effect triggering operation setting unit.

The special effect triggering operation setting unit is used for detecting that a visual field area comprises a target object, triggering target special effect actions by detecting the target object and triggering special effect generation controls.

Based on the above technical solutions, the target special effect image obtaining module 720 includes a target condition label determining unit and a target special effect map obtaining unit.

A target condition label determining unit for determining at least one target condition label corresponding to the image to be processed;

and the target special effect diagram acquisition unit is used for taking the at least one target condition label and the image to be processed as the input of the target pupil adjustment model to obtain the target special effect diagram.

On the basis of the above technical solutions, the target condition tag determining unit includes a target condition tag obtaining subunit.

The target condition label acquisition subunit is used for acquiring at least one preset target condition label or determining at least one target condition label corresponding to the target object according to the gesture information of the target object in the image to be processed.

On the basis of the above technical solutions, the target condition tag determining unit further includes a target condition tag setting subunit.

The target condition label setting subunit is configured to set at least one target condition label to include a target size label corresponding to a target display size of the pupil, and/or a target position label corresponding to a target relative display position of the pupil in the eye.

On the basis of the above technical solutions, the target condition tag setting subunit includes that the target size tag includes a first unit, the target position tag includes a first unit and the target position, and the size tag includes a first unit.

The target size label comprises a first unit, wherein the first unit is used for obtaining the target pupil characteristic adjustment model through training under the conditions of fixed position labels and variable size labels, and the at least one target condition label comprises a target size label;

The target position label comprises a first unit, wherein the first unit is used for including a target position label in at least one target condition label if the target pupil characteristic adjustment model is obtained by training under the conditions of fixed size label and position label transformation;

The target position and size label comprises a first unit, and the first unit is used for including a target position label and a target size label in the at least one target condition label if the target pupil characteristic adjustment model is obtained through training under the condition that the size label and the position label are changed.

On the basis of the above technical solutions, the target size tag includes a first unit, and the target size tag includes a first subunit. The target location tag includes a first unit, including the target location tag including a first subunit. The target location, size label includes a first unit including the target location, and the size label includes a first subunit.

The target size label comprises a first subunit, and is configured to adjust, based on the target pupil characteristic adjustment model, the pupil of the target object in the image to be processed to match with the target size label and a position label used when the target pupil characteristic adjustment model is obtained by training, if the at least one target condition label comprises the target size label;

the target position label comprises a first subunit, and is configured to adjust, based on the target pupil characteristic adjustment model, the pupil of the target object in the image to be processed to match with the target position label and a size label used when the target pupil characteristic adjustment model is obtained by training, if the at least one target condition label comprises the target position label;

And the target position and size label comprises a first subunit, and the first subunit is used for adjusting the pupil of the target object in the image to be processed to be matched with the target size label and the target position label based on the target pupil characteristic adjustment model if the at least one target condition label comprises the target size label and the target position label, so as to obtain the target special effect diagram.

Based on the technical schemes, the device further comprises a target pupil characteristic adjustment model acquisition module. The target pupil characteristic adjustment model acquisition module comprises a first training sample set acquisition unit, a model parameter adjustment unit and a target pupil characteristic adjustment model acquisition unit.

The first training sample set acquisition unit is used for acquiring a first training sample set, wherein the first training sample set comprises a plurality of first training samples, the first training samples comprise a first original input image and a first theoretical image, and the first theoretical image is generated based on a target pupil position adjustment model and a target pupil size adjustment model which are obtained through pre-training;

the model parameter adjusting unit is used for regarding each first training sample, taking a first original image of a current first training sample as input of a pupil characteristic adjusting model to be trained, taking the first theoretical image as output of the pupil characteristic adjusting model to be trained, and adjusting model parameters of the pupil characteristic adjusting model to be trained;

and the target pupil characteristic adjustment model acquisition unit is used for converging the loss function of the pupil characteristic adjustment model to be trained as a training target to obtain the target pupil characteristic adjustment model.

On the basis of the above technical solutions, the first training sample set obtaining unit includes a first original input image obtaining subunit, an image obtaining subunit to be used, and a first theoretical image obtaining subunit.

A first original input image acquisition subunit configured to acquire a first original input image;

The to-be-used image acquisition subunit is used for inputting the first original input image and the pupil position label to be displayed into the target pupil position adjustment model to obtain an to-be-used image;

the first theoretical image acquisition subunit is used for inputting the image to be used and the pupil size label to be displayed into the target pupil size adjustment model to obtain the first theoretical image.

On the basis of the above technical solutions, the first training sample set obtaining unit further includes a fixed value first subunit, a fixed value second subunit, and a label changing third subunit.

The fixed value first subunit is configured to, if the pupil size label to be displayed is a fixed value, make each first theoretical image be an image with the same pupil size and different pupil positions;

The fixed value second subunit is used for if the pupil position label to be displayed is a fixed value, each first theoretical image is an image with the same pupil position and different pupil sizes;

And the label change third subunit is configured to, if the pupil size label to be displayed changes, change the pupil position label to be displayed, where each first theoretical image is an image with a difference between a pupil position and a pupil size.

On the basis of the technical schemes, the device further comprises a target pupil position adjustment model acquisition module. The target pupil position adjustment model acquisition module comprises a second training sample set acquisition unit, a model parameter adjustment unit and a target pupil position adjustment model acquisition unit.

The system comprises a first training sample set acquisition unit, a second training sample set acquisition unit and a target pupil reconstruction model, wherein the first training sample set comprises a plurality of first training samples, the first training samples comprise a first input image, a first theoretical output image and a pupil position label corresponding to the first theoretical output image, and the first input image and the first theoretical output image are determined based on a target pupil reconstruction model obtained through training in advance;

The model parameter adjusting unit is used for regarding each second training sample, taking a second input image and a pupil position label of the current training sample as input of a pupil position adjusting model to be trained, taking a second theoretical output image as output of the pupil position adjusting model to be trained, and adjusting model parameters of the pupil position adjusting model to be trained;

And the target pupil position adjustment model acquisition unit is used for converging the loss function of the pupil position adjustment model to be trained as a training target to obtain the target pupil position adjustment model.

On the basis of the technical schemes, the device further comprises a target pupil size adjustment model acquisition module. The target pupil size adjustment model acquisition module comprises a third training sample set acquisition unit, a model parameter adjustment unit and a target pupil size adjustment model acquisition unit.

The system comprises a first training sample set acquisition unit, a second training sample set acquisition unit and a first training sample set acquisition unit, wherein the first training sample set comprises a plurality of first training samples, the first training samples comprise a first input image, a first theoretical output image and a first pupil size label corresponding to the first theoretical image, and the first input image and the first theoretical output image are determined based on a target pupil reconstruction model obtained through training in advance;

The model parameter adjusting unit is used for regarding each third training sample, taking a third input image and a pupil size label of the current training sample as input of a pupil position adjusting model to be trained, taking a third theoretical output image as output of the pupil position adjusting model to be trained, and adjusting model parameters of the pupil position adjusting model to be trained;

And the target pupil size adjustment model acquisition unit is used for converging the loss function of the pupil size adjustment model to be trained as a training target to obtain the target pupil size adjustment model.

Based on the technical schemes, the device further comprises a target pupil reconstruction model acquisition module. The target pupil reconstruction model acquisition module comprises a fourth training sample set acquisition unit, a model parameter adjustment unit and a target pupil reconstruction model acquisition unit.

The system comprises a fourth training sample set acquisition unit, a first training sample set acquisition unit and a second training sample set acquisition unit, wherein the fourth training sample set acquisition unit is used for determining a plurality of fourth training samples, the fourth training samples comprise a preprocessed image, and an image to be used, which comprises pupil marks, in the preprocessed image is determined according to a preset image processing algorithm;

The model parameter adjusting unit is used for regarding each fourth training sample, taking the image to be used as an input parameter of a pupil reconstruction model to be trained, taking the preprocessed image as an output parameter of the pupil reconstruction model to be trained, and adjusting the model parameters of the pupil reconstruction model to be trained;

And the target pupil reconstruction model acquisition unit is used for converging the loss function of the pupil reconstruction model to be trained as a training target to obtain the target pupil reconstruction model.

On the basis of the above technical solutions, the fourth training sample set obtaining unit includes a preprocessing image obtaining subunit, a region determining subunit, and an image obtaining subunit to be used.

A preprocessing image acquisition subunit, configured to acquire a preprocessing image including a target portion, and determine a gray level map of the target portion;

The region determining subunit is used for determining a first region and a second region in the gray scale image according to a preset image processing algorithm, wherein the target part is an eye part, the first region is a pupil region, and the second region is an eye white region;

And the image to be used acquisition subunit is used for acquiring an image to be used including pupil marks based on the first area and the second area.

On the basis of the technical schemes, the region determining subunit comprises a region determining first unit, a region determining second unit and a region determining third unit.

The preset image processing algorithm comprises a gray average value algorithm. The first unit for determining the region comprises a first subunit for determining the gray average value, a first subunit for determining the pixel point and a first subunit for determining the region.

A gray average value determining first subunit, configured to determine a gray average value of the gray map;

The pixel point determining first subunit is configured to use a pixel point lower than the gray average value as a first type pixel point and a pixel point higher than the gray average value as a second type pixel point;

And the region determining first subunit is used for determining a first region and a second region based on the first type pixel point and the second type pixel point.

The preset image processing algorithm comprises a gray threshold algorithm. And the region determining second unit comprises a pixel point determining second subunit and a region determining second subunit.

The pixel point determining second subunit is configured to determine a first type pixel point of a pixel point with a gray value smaller than a preset gray value threshold, and use a pixel point with a gray value greater than or equal to the preset gray value threshold as a second type pixel point;

And the region determining second subunit is used for determining a first region and a second region based on the first type pixel point and the second type pixel point.

On the basis of the technical schemes, the region determining third unit comprises a first segmented image determining third subunit, a second segmented image determining third subunit, a pixel point intersection set determining third subunit and a region determining third subunit.

The first segmentation image determining unit is used for determining a first pixel point to be determined, the gray average value of which is larger than a preset gray value threshold value, and determining a second pixel point to be determined, the gray value of which is smaller than the preset gray value threshold value, and determining a first segmentation image based on the first pixel point to be determined and the second pixel point to be determined;

a third subunit for determining a third to-be-determined pixel point with a gray average value larger than the gray average value of the target part, determining a fourth to-be-determined pixel point with a gray average value smaller than the gray average value, and determining a second divided image based on the third to-be-determined pixel point and the fourth to-be-determined pixel point;

A third subunit, configured to determine a pixel intersection and a pixel union according to a first to-be-determined pixel and the third to-be-determined pixel in the first segmented image and the second segmented image;

And the region determining third subunit is configured to determine, if a ratio of the intersection of the pixel points to the union of the pixel points is greater than a preset ratio threshold, a first region and a second region based on the intersection of the pixel points.

According to the technical scheme, the to-be-processed image comprising the target object is acquired through responding to the special effect triggering operation, the to-be-processed image is processed based on the target pupil characteristic adjustment model, and at least one target condition label corresponding to the to-be-processed image is processed to obtain the target special effect image, the target special effect image is displayed on the target display interface, the problem that the reality of the obtained special effect image is low when the special effect image is generated by using the picture repairing technology in the prior art is solved, so that poor user experience is caused is solved, the pupil is finely adjusted based on the pupil attribute condition label, special effects are added to the eyes, the generated special effect image is consistent with the condition label, the accuracy of adding the eye special effects is improved, and further when the special effect image is displayed on the display interface, the displayed image is more vivid, and the technical effect of meeting the user experience requirement is achieved.

The image processing device provided by the embodiment of the disclosure can execute the image processing method provided by any embodiment of the disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

It should be noted that the above-mentioned units and modules included in the apparatus are only divided according to the functional logic, but not limited to the above-mentioned division, so long as the corresponding functions can be implemented, and the specific names of the functional units are only used for distinguishing from each other, and are not used for limiting the protection scope of the embodiments of the present disclosure.

Example eight

Fig. 12 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure. Referring now to fig. 12, a schematic diagram of an electronic device (e.g., a terminal device or server in fig. 12) 800 suitable for use in implementing embodiments of the present disclosure is shown. The terminal devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and the like, and stationary terminals such as digital TVs, desktop computers, and the like. The electronic device shown in fig. 12 is merely an example and should not be construed to limit the functionality and scope of use of the disclosed embodiments.

As shown in fig. 12, the electronic device 800 may include a processing means (e.g., a central processor, a graphics processor, etc.) 801, which may perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 802 or a program loaded from a storage means 808 into a Random Access Memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 800 are also stored. The processing device 801, the ROM 802, and the RAM 803 are connected to each other by a bus 804. An input/output (I/O) interface 805 is also connected to the bus 804.

In general, devices may be connected to I/O interface 805 including input devices 806 such as a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc., output devices 807 including a Liquid Crystal Display (LCD), speaker, vibrator, etc., storage devices 808 including magnetic tape, hard disk, etc., and communication devices 809. The communication means 809 may allow the electronic device 800 to communicate wirelessly or by wire with other devices to exchange data. While fig. 12 shows an electronic device 800 having various means, it is to be understood that not all illustrated means are required to be implemented or provided. More or fewer devices may be implemented or provided instead.

In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a non-transitory computer readable medium, the computer program comprising program code for performing the method shown in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via communication device 809, or installed from storage device 808, or installed from ROM 802. The above-described functions defined in the methods of the embodiments of the present disclosure are performed when the computer program is executed by the processing device 801.

The names of messages or information interacted between the various devices in the embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

The electronic device provided by the embodiment of the present disclosure and the image processing method provided by the foregoing embodiment belong to the same disclosure concept, and technical details not described in detail in the present embodiment may be referred to the foregoing embodiment, and the present embodiment has the same beneficial effects as the foregoing embodiment.

Example nine

A ninth embodiment of the present disclosure provides a computer storage medium having stored thereon a computer program which, when executed by a processor, implements the image processing method provided by the above embodiment.

It should be noted that the computer readable medium described in the present disclosure may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of a computer-readable storage medium may include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this disclosure, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present disclosure, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to electrical wiring, fiber optic cable, RF (radio frequency), and the like, or any suitable combination of the foregoing.

In some embodiments, the clients, servers may communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol ), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), the internet (e.g., the internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed networks.

The computer readable medium may be included in the electronic device or may exist alone without being incorporated into the electronic device.

The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to:

responding to the special effect triggering operation, and collecting an image to be processed comprising a target object;

Processing the image to be processed and at least one target condition label corresponding to the image to be processed based on a target pupil characteristic adjustment model to obtain a target special effect image, wherein pupil characteristics of a target object in the target special effect image are matched with the at least one target condition label;

and displaying the target special effect image on a target display interface.

Computer program code for carrying out operations of the present disclosure may be written in one or more programming languages, including, but not limited to, an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The units involved in the embodiments of the present disclosure may be implemented by means of software, or may be implemented by means of hardware. The name of the unit does not in any way constitute a limitation of the unit itself, for example the first acquisition unit may also be described as "unit acquiring at least two internet protocol addresses".

The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic that may be used include Field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems-on-a-chip (SOCs), complex Programmable Logic Devices (CPLDs), and the like.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

According to one or more embodiments of the present disclosure, there is provided an image processing method, the method including:

responding to the special effect triggering operation, and collecting an image to be processed comprising a target object;

Processing the image to be processed and at least one target condition label corresponding to the image to be processed based on a target pupil characteristic adjustment model to obtain a target special effect image, wherein pupil characteristics of a target object in the target special effect image are matched with the at least one target condition label;

and displaying the target special effect image on a target display interface.

According to one or more embodiments of the present disclosure, there is provided an image processing method [ example two ] further including:

optionally, the special effect triggering operation includes at least one of the following:

detecting that a target object is included in the field of view;

detecting that the target object triggers a target special effect action;

the trigger special effect generation control is detected.

According to one or more embodiments of the present disclosure, there is provided an image processing method [ example three ], further comprising:

Optionally, determining at least one target condition label corresponding to the image to be processed;

and taking the at least one target condition label and the image to be processed as input of the target pupil adjustment model to obtain the target special effect diagram.

According to one or more embodiments of the present disclosure, there is provided an image processing method [ example four ], further comprising:

acquiring at least one preset target condition label, or,

And determining at least one target condition label corresponding to the target object according to the gesture information of the target object in the image to be processed.

According to one or more embodiments of the present disclosure, there is provided an image processing method [ example five ]:

Optionally, the at least one target condition label includes a target size label corresponding to a target display size of the pupil, and/or a target position label corresponding to a target relative display position of the pupil in the eye.

According to one or more embodiments of the present disclosure, there is provided an image processing method [ example six ], further comprising:

If the target pupil characteristic adjustment model is obtained by training under the condition of fixed position labels and variable size labels, the at least one target condition label comprises a target size label;

If the target pupil characteristic adjustment model is obtained by training under the condition of fixed size labels and variable position labels, the at least one target condition label comprises a target position label;

And if the target pupil characteristic adjustment model is obtained by training under the condition of size label and position label transformation, the at least one target condition label comprises a target position label and a target size label.

According to one or more embodiments of the present disclosure, there is provided an image processing method [ example seventh ], further comprising:

If the at least one target condition label comprises a target size label and a target position label, based on the target pupil characteristic adjustment model, adjusting the pupil of a target object in the image to be processed to be matched with the target size label and the target position label, and obtaining the target special effect diagram;

If the at least one target condition label comprises a target size label, based on the target pupil characteristic adjustment model, adjusting the pupil of a target object in the image to be processed to be matched with the target size label and a position label used when the target pupil characteristic adjustment model is obtained through training;

And if the at least one target condition label comprises a target position label, adjusting the pupil of the target object in the image to be processed to be matched with the target position label and a size label used when the target pupil characteristic adjustment model is obtained through training based on the target pupil characteristic adjustment model.

According to one or more embodiments of the present disclosure, there is provided an image processing method [ example eight ], further comprising:

training to obtain the target pupil characteristic adjustment model;

the training to obtain the target pupil characteristic adjustment model comprises the following steps:

The method comprises the steps of obtaining a first training sample set, wherein the first training sample set comprises a plurality of first training samples, the first training samples comprise a first original input image and a first theoretical image, and the first theoretical image is generated based on a target pupil position adjustment model and a target pupil size adjustment model which are obtained through training in advance;

Aiming at each first training sample, taking a first original image of a current first training sample as input of a pupil characteristic adjustment model to be trained, taking the first theoretical image as output of the pupil characteristic adjustment model to be trained, and adjusting model parameters of the pupil characteristic adjustment model to be trained;

and converging the loss function of the pupil characteristic adjustment model to be trained as a training target to obtain the target pupil characteristic adjustment model.

According to one or more embodiments of the present disclosure, there is provided an image processing method, further including:

acquiring a first original input image;

inputting the first original input image and the pupil position label to be displayed into the target pupil position adjustment model to obtain an image to be used;

And inputting the image to be used and the pupil size label to be displayed into the target pupil size adjustment model to obtain the first theoretical image.

According to one or more embodiments of the present disclosure, there is provided an image processing method, further comprising:

if the pupil size label to be displayed is a fixed value, each first theoretical image is an image with the same pupil size and different pupil positions;

if the pupil position label to be displayed is a fixed value, the first theoretical images are images with the same pupil position and different pupil sizes;

If the pupil size label to be displayed changes, and the pupil position label to be displayed changes, each first theoretical image is an image with difference between the pupil position and the pupil size.

According to one or more embodiments of the present disclosure, there is provided an image processing method [ example eleven ], further comprising:

training to obtain the target pupil position adjustment model:

the training obtains the target pupil position adjustment model, which comprises the following steps:

The method comprises the steps of obtaining a first training sample set, wherein the first training sample set comprises a plurality of first training samples, the first training samples comprise a first input image, a first theoretical output image and a first pupil position label corresponding to the first theoretical output image, the first input image and the first theoretical output image are determined based on a target pupil reconstruction model obtained through training in advance, and the target pupil reconstruction model is used for reconstructing an image consistent with a marked pupil;

Aiming at each second training sample, taking a second input image and a pupil position label of the current training sample as input of a pupil position adjustment model to be trained, taking a second theoretical output image as output of the pupil position adjustment model to be trained, and adjusting model parameters of the pupil position adjustment model to be trained;

And converging the loss function of the pupil position adjustment model to be trained as a training target to obtain the target pupil position adjustment model.

According to one or more embodiments of the present disclosure, there is provided an image processing method [ example twelve ], further comprising:

Training to obtain a target pupil size adjustment model;

The training to obtain a target pupil size adjustment model includes:

The method comprises the steps of obtaining a third training sample set, wherein the third training sample set comprises a plurality of third training samples, the third training samples comprise a third input image, a third theoretical output image and pupil size labels corresponding to the third theoretical image, and the third input image and the third theoretical output image are determined based on a target pupil reconstruction model obtained through pre-training;

aiming at each third training sample, taking a third input image and a pupil size label of the current training sample as input of a pupil position adjustment model to be trained, taking a third theoretical output image as output of the pupil position adjustment model to be trained, and adjusting model parameters of the pupil position adjustment model to be trained;

And converging the loss function of the pupil size adjustment model to be trained as a training target to obtain the target pupil size adjustment model.

According to one or more embodiments of the present disclosure, there is provided an image processing method [ example thirteenth ], further comprising:

Training to obtain a target pupil reconstruction model;

the training to obtain a target pupil reconstruction model includes:

determining a plurality of fourth training samples, wherein the fourth training samples comprise preprocessed images, and determining images to be used, which comprise pupil marks, in the preprocessed images according to a preset image processing algorithm;

Aiming at each fourth training sample, taking the image to be used as an input parameter of a pupil reconstruction model to be trained, taking the preprocessed image as an output parameter of the pupil reconstruction model to be trained, and adjusting model parameters of the pupil reconstruction model to be trained;

and converging the loss function of the pupil reconstruction model to be trained as a training target to obtain the target pupil reconstruction model.

According to one or more embodiments of the present disclosure, there is provided an image processing method [ example fourteen ], further comprising:

acquiring a preprocessing image comprising a target part, and determining a gray level image of the target part;

Determining a first area and a second area in the gray scale image according to a preset image processing algorithm, wherein the target part is an eye part, the first area is a pupil area, and the second area is an eye white area;

And obtaining an image to be used including pupil marks based on the first area and the second area.

According to one or more embodiments of the present disclosure, there is provided an image processing method [ example fifteen ], further comprising:

optionally, the preset image processing algorithm includes a gray-scale average algorithm;

Determining a gray average value of the gray map;

taking the pixel points lower than the gray average value as first type pixel points, and taking the pixel points higher than the gray average value as second type pixel points;

And determining a first area and a second area based on the first type pixel point and the second type pixel point.

According to one or more embodiments of the present disclosure, there is provided an image processing method [ example sixteen ], further comprising:

optionally, the preset image processing algorithm includes a gray threshold algorithm.

Taking the pixel points with the gray values smaller than the preset gray value threshold value as the first type pixel points and the pixel points with the gray values larger than or equal to the preset gray value threshold value as the second type pixel points;

And determining a first area and a second area based on the first type pixel point and the second type pixel point.

According to one or more embodiments of the present disclosure, there is provided an image processing method [ example seventeen ], further comprising:

determining a first pixel to be determined, the gray average value of which is larger than a preset gray value threshold value, and determining a second pixel to be determined, the gray value of which is smaller than the preset gray value threshold value, and determining a first segmentation image based on the first pixel to be determined and the second pixel to be determined;

Determining a third pixel point to be determined, the gray average value of which is larger than the gray average value of the target part, determining a fourth pixel point to be determined, the gray average value of which is smaller than the gray average value of the gray average value, and determining a second segmentation image based on the third pixel point to be determined and the fourth pixel point to be determined;

Determining a pixel point intersection and a pixel point union according to a first pixel point to be determined and the third pixel point to be determined in the first segmentation image and the second segmentation image;

And if the ratio of the pixel point intersection to the pixel point union is greater than a preset ratio threshold, determining a first area and a second area based on the pixel point intersection.

According to one or more embodiments of the present disclosure, there is provided an image processing apparatus, including:

The image acquisition module to be processed is used for responding to the special effect triggering operation and acquiring an image to be processed comprising the target object;

The target special effect image acquisition module is used for processing the image to be processed and at least one target condition label corresponding to the image to be processed based on a target pupil characteristic adjustment model to obtain a target special effect image, wherein pupil characteristics of a target object in the target special effect image are matched with the at least one target condition label;

And the image display module is used for displaying the target special effect image on a target display interface.

The foregoing description is only of the preferred embodiments of the present disclosure and description of the principles of the technology being employed. It will be appreciated by persons skilled in the art that the scope of the disclosure referred to in this disclosure is not limited to the specific combinations of features described above, but also covers other embodiments which may be formed by any combination of features described above or equivalents thereof without departing from the spirit of the disclosure. Such as those described above, are mutually substituted with the technical features having similar functions disclosed in the present disclosure (but not limited thereto).

Moreover, although operations are depicted in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order. In certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the present disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.

Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are example forms of implementing the claims.

Claims (19)

1. An image processing method, comprising:

responding to the special effect triggering operation, and collecting an image to be processed comprising a target object;

Processing the image to be processed and at least one target condition label corresponding to the image to be processed based on a target pupil characteristic adjustment model to obtain a target special effect image, wherein pupil characteristics of a target object in the target special effect image are matched with the at least one target condition label;

displaying the target special effect image on a target display interface;

The method further comprises the step of if the target pupil characteristic adjustment model is obtained through training under the conditions of fixed position labels and variable size labels, then the at least one target condition label comprises a target size label.

2. The method of claim 1, wherein the special effect triggering operation comprises at least one of:

detecting that a target object is included in the field of view;

detecting that the target object triggers a target special effect action;

the trigger special effect generation control is detected.

3. The method of claim 1, wherein the processing the image to be processed and the at least one target condition label corresponding to the image to be processed based on the target pupil characteristic adjustment model to obtain a target special effect image comprises:

Determining at least one target condition label corresponding to the image to be processed;

and taking the at least one target condition label and the image to be processed as input of the target pupil adjustment model to obtain the target special effect diagram.

4. A method according to claim 3, wherein said determining at least one target condition label corresponding to said image to be processed comprises:

acquiring at least one preset target condition label, or,

And determining at least one target condition label corresponding to the target object according to the gesture information of the target object in the image to be processed.

5. The method as recited in claim 1, further comprising:

If the target pupil characteristic adjustment model is obtained by training under the condition of fixed size labels and variable position labels, the at least one target condition label comprises a target position label;

And if the target pupil characteristic adjustment model is obtained by training under the condition of size label and position label transformation, the at least one target condition label comprises a target position label and a target size label.

6. The method according to claim 1 or 5, wherein the processing the image to be processed and at least one target condition label corresponding to the image to be processed based on the target pupil characteristic adjustment model to obtain a target special effect image includes:

If the at least one target condition label comprises a target size label and a target position label, based on the target pupil characteristic adjustment model, adjusting the pupil of a target object in the image to be processed to be matched with the target size label and the target position label, and obtaining the target special effect diagram;

If the at least one target condition label comprises a target size label, based on the target pupil characteristic adjustment model, adjusting the pupil of a target object in the image to be processed to be matched with the target size label and a position label used when the target pupil characteristic adjustment model is obtained through training;

And if the at least one target condition label comprises a target position label, adjusting the pupil of the target object in the image to be processed to be matched with the target position label and a size label used when the target pupil characteristic adjustment model is obtained through training based on the target pupil characteristic adjustment model.

7. The method as recited in claim 1, further comprising:

training to obtain the target pupil characteristic adjustment model;

the training to obtain the target pupil characteristic adjustment model comprises the following steps:

The method comprises the steps of obtaining a first training sample set, wherein the first training sample set comprises a plurality of first training samples, the first training samples comprise a first original input image and a first theoretical image, and the first theoretical image is generated based on a target pupil position adjustment model and a target pupil size adjustment model which are obtained through training in advance;

Aiming at each first training sample, taking a first original image of a current first training sample as input of a pupil characteristic adjustment model to be trained, taking the first theoretical image as output of the pupil characteristic adjustment model to be trained, and adjusting model parameters of the pupil characteristic adjustment model to be trained;

and converging the loss function of the pupil characteristic adjustment model to be trained as a training target to obtain the target pupil characteristic adjustment model.

8. The method of claim 7, wherein determining a first theoretical image in each first training sample in the first training sample set comprises:

acquiring a first original input image;

inputting the first original input image and the pupil position label to be displayed into the target pupil position adjustment model to obtain an image to be used;

And inputting the image to be used and the pupil size label to be displayed into the target pupil size adjustment model to obtain the first theoretical image.

9. The method as recited in claim 8, further comprising:

if the pupil size label to be displayed is a fixed value, each first theoretical image is an image with the same pupil size and different pupil positions;

if the pupil position label to be displayed is a fixed value, the first theoretical images are images with the same pupil position and different pupil sizes;

If the pupil size label to be displayed changes, and the pupil position label to be displayed changes, each first theoretical image is an image with difference between the pupil position and the pupil size.

10. The method as recited in claim 7, further comprising:

Training to obtain the target pupil position adjustment model;

the training obtains the target pupil position adjustment model, which comprises the following steps:

The method comprises the steps of obtaining a first training sample set, wherein the first training sample set comprises a plurality of first training samples, the first training samples comprise a first input image, a first theoretical output image and a first pupil position label corresponding to the first theoretical output image, the first input image and the first theoretical output image are determined based on a target pupil reconstruction model obtained through training in advance, and the target pupil reconstruction model is used for reconstructing an image consistent with a marked pupil;

Aiming at each second training sample, taking a second input image and a pupil position label of the current training sample as input of a pupil position adjustment model to be trained, taking a second theoretical output image as output of the pupil position adjustment model to be trained, and adjusting model parameters of the pupil position adjustment model to be trained;

And converging the loss function of the pupil position adjustment model to be trained as a training target to obtain the target pupil position adjustment model.

11. The method as recited in claim 7, further comprising:

Training to obtain a target pupil size adjustment model;

The training to obtain a target pupil size adjustment model includes:

The method comprises the steps of obtaining a third training sample set, wherein the third training sample set comprises a plurality of third training samples, the third training samples comprise a third input image, a third theoretical output image and pupil size labels corresponding to the third theoretical image, and the third input image and the third theoretical output image are determined based on a target pupil reconstruction model obtained through pre-training;

aiming at each third training sample, taking a third input image and a pupil size label of the current training sample as input of a pupil position adjustment model to be trained, taking a third theoretical output image as output of the pupil position adjustment model to be trained, and adjusting model parameters of the pupil position adjustment model to be trained;

And converging the loss function of the pupil size adjustment model to be trained as a training target to obtain the target pupil size adjustment model.

12. The method according to claim 10 or 11, further comprising:

Training to obtain a target pupil reconstruction model;

the training to obtain a target pupil reconstruction model includes:

determining a plurality of fourth training samples, wherein the fourth training samples comprise preprocessed images, and determining images to be used, which comprise pupil marks, in the preprocessed images according to a preset image processing algorithm;

Aiming at each fourth training sample, taking the image to be used as an input parameter of a pupil reconstruction model to be trained, taking the preprocessed image as an output parameter of the pupil reconstruction model to be trained, and adjusting model parameters of the pupil reconstruction model to be trained;

and converging the loss function of the pupil reconstruction model to be trained as a training target to obtain the target pupil reconstruction model.

13. The method of claim 12, wherein determining each fourth training sample comprises:

acquiring a preprocessing image comprising a target part, and determining a gray level image of the target part;

Determining a first area and a second area in the gray scale image according to a preset image processing algorithm, wherein the target part is an eye part, the first area is a pupil area, and the second area is an eye white area;

And obtaining an image to be used including pupil marks based on the first area and the second area.

14. The method of claim 13, wherein the predetermined image processing algorithm comprises a gray-scale average algorithm, and wherein determining the first region and the second region in the gray-scale map according to the predetermined image processing algorithm comprises:

Determining a gray average value of the gray map;

taking the pixel points lower than the gray average value as first type pixel points, and taking the pixel points higher than the gray average value as second type pixel points;

And determining a first area and a second area based on the first type pixel point and the second type pixel point.

15. The method of claim 13, wherein the predetermined image processing algorithm comprises a gray threshold algorithm, the determining the first region and the second region in the gray scale map according to the predetermined image processing algorithm comprising:

taking the pixel points with the gray values smaller than the preset gray value threshold value as the first type pixel points and the pixel points with the gray values larger than or equal to the preset gray value threshold value as the second type pixel points;

And determining a first area and a second area based on the first type pixel point and the second type pixel point.

16. The method of claim 13, wherein determining the first region and the second region in the gray scale map according to a preset image processing algorithm comprises:

determining a first pixel to be determined, the gray average value of which is larger than a preset gray value threshold value, and determining a second pixel to be determined, the gray value of which is smaller than the preset gray value threshold value, and determining a first segmentation image based on the first pixel to be determined and the second pixel to be determined;

Determining a third pixel point to be determined, the gray average value of which is larger than the gray average value of the target part, determining a fourth pixel point to be determined, the gray average value of which is smaller than the gray average value of the gray average value, and determining a second segmentation image based on the third pixel point to be determined and the fourth pixel point to be determined;

Determining a pixel point intersection and a pixel point union according to a first pixel point to be determined and the third pixel point to be determined in the first segmentation image and the second segmentation image;

And if the ratio of the pixel point intersection to the pixel point union is greater than a preset ratio threshold, determining a first area and a second area based on the pixel point intersection.

17. An image processing apparatus, comprising:

The image acquisition module to be processed is used for responding to the special effect triggering operation and acquiring an image to be processed comprising the target object;

The target special effect image acquisition module is used for processing the image to be processed and at least one target condition label corresponding to the image to be processed based on a target pupil characteristic adjustment model to obtain a target special effect image, wherein the pupil characteristic of a target object in the target special effect image is matched with the at least one target condition label;

The image display module is used for displaying the target special effect image on a target display interface;

The device further comprises a target condition label setting subunit, wherein the target condition label setting subunit is used for setting at least one target condition label to comprise a target size label if the target pupil characteristic adjustment model is obtained by training under the conditions of fixed position labels and size label transformation.

18. An electronic device, the electronic device comprising:

One or more processors;

storage means for storing one or more programs,

When executed by the one or more processors, causes the one or more processors to implement the image processing method of any of claims 1-16.

19. A storage medium containing computer executable instructions which, when executed by a computer processor, are for performing the image processing method of any of claims 1-16.