Detailed Description
Exemplary embodiments of the present application are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present application to facilitate understanding, and should be considered as merely exemplary. Accordingly, one of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present application. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
According to the image processing method, a first target image block is selected from a first image to be processed, then an image block closest to the first target image block is searched in a second image to be a second target image block, and then the content in the second target image block is migrated to a target area in the first target image block.
The embodiment of the application first provides an image processing method, as shown in fig. 1, including:
step 101: a target region in the first image is determined.
In the embodiment of the present application, the target area may be acquired according to preset information. For example, the first image is a facial image, the makeup regions in the facial image may include an eye makeup region, a cheek makeup region, a lip makeup region, a eyebrow makeup region, and the target region may be at least one of the makeup regions. Alternatively, the first image is a facial image including facial organs such as eyes, cheeks, lips, eyebrows, etc., and the target area is at least one of the facial organ areas.
In one embodiment of the present application, the first image is a facial image and the target area is one of the make-up areas. The application area of different ranges can be selected and determined according to the application type. For example, the makeup type may be light makeup, heavy makeup, bright makeup, beijing opera facial makeup, etc. The type of makeup may be determined based on a user selected parameter or may be determined based on the first image. For example, if the area occupied by the gorgeous color in the first image is large, the makeup type can be determined to be gorgeous. The type of makeup may also be determined from the second image. For example, the second image is mostly light or dark, and the makeup type may be determined to be light. Different makeup types, corresponding makeup ranges and target areas are different. For example, the range of the fumigated makeup corresponding to the eyes is larger, and the target area is larger; the light make-up corresponds to a smaller range of make-up on the eyes and a smaller corresponding target area.
The target region may be acquired according to the type of the first image. For example, in one embodiment, the first image is a facial image and the target area is a make-up area in the facial image. In another embodiment, the first image may be other images, such as a building image, an animal face image, a sculptured face image, etc., and the target area is a preset area where the content needs to be changed.
Step 102: and determining a first target image block comprising the target area in the first image according to the target area.
In an embodiment of the present application, the first image block may coincide with the target area.
In another embodiment of the present application, the first target image block is a circumscribed rectangle of the target area. For example, the first image is a facial image, the target area may be one of an eye make-up area, an eyebrow make-up area, a cheek make-up area, and a lip make-up area, and the target area may be an irregular pattern. In this case, the first target image block may be a rectangle circumscribing the target area pattern.
For another example, the first image is a facial image, the target area may be one of an eye area, a eyebrow area, a cheek area, and a lip area, and the first target image block may be a rectangle circumscribing the target area pattern.
Step 103: and determining a second target image block in the second image according to the first target image block.
In the embodiment of the present application, the second target image block may be an image block (Patch) in the second image, in which a relationship between the second target image block and the first target image block conforms to a set value. For example, the second target image block may be an image block in the second image having an image characteristic closest to the first target image block.
Step 104: and fusing the content in the second target image block with the content in the first target image block in the target area.
In the embodiment of the application, the content in the second target image block is fused with the content in the first target image block, and the content in the second target image block may be selectively displayed in the target area in the first target image block.
In the embodiment of the application, the content in the second target image block is fused with the content in the first target image block, and part or all of the image features in the first target image block can be changed by using part or all of the image features in the second target image block.
For the first image, where there may be multiple target areas, the operations of steps 101-104 described above may be performed on each target area one by one.
In the embodiment of the application, the second target image block can be determined in the second image according to the target area in the first image, and then the content in the second target image block is fused with the content in the first target image block in the target area, so that the technical effects of processing and adjusting the first image according to the elements in the second image can be realized. In practical application, the method can be applied to scenes and the like for making up the face of the user by referring to the pictures, and provides diversified processing modes for realizing applications such as terminal image beautification or video beautification.
In one embodiment, as shown in fig. 2, in step S104 may include:
step 201: determining a target mask layer corresponding to the target region from a plurality of preset mask layers, wherein the target mask layer comprises a first presentation proportion and a second presentation proportion;
step 202: and in the target area, the corresponding content of the first target image block is presented in a first presentation proportion, and the corresponding content of the second target image block is presented in a second presentation proportion.
In this embodiment of the present application, the first rendering ratio and the second rendering ratio may be rendering ratios for all pixels in the target area, or may be rendering ratios for some pixels in the target area.
In one embodiment, the sum of the first presentation scale and the second presentation scale is 1. For example, 30% of the first target image block content is presented and 70% of the second target image block content is presented. Therefore, the purpose of content migration can be achieved, and characteristics such as textures in the first target image block can be reserved.
In this embodiment of the present application, the plurality of preset mask layers may correspond to different target region types. For example, the first image is a facial image, and if the target region is an eye makeup region, the mask layer is a mask layer corresponding to the eye makeup region.
In the embodiment of the present invention, in the target area, the content in the first target image block is presented according to the set first presentation ratio, and the content in the second target image block is presented according to the set second presentation ratio, so that the content in the second target image block of the second image can be migrated to the target area of the first image. Thus, the first image has the content and style of the second image, and a new function is developed for video and image shooting of the current terminal.
In one embodiment, as shown in fig. 3, in step S104 may include:
step 301: determining a target mask layer corresponding to a target area from a plurality of preset mask layers, wherein the target mask layer comprises a first presentation proportion and a second presentation proportion of each pixel in the target area;
step 302: for each pixel in the target area, the corresponding content of the first target image block is presented on the pixel at a first presentation scale corresponding to the pixel, and the corresponding content of the second target image block is presented at a second presentation scale corresponding to the pixel.
In the embodiment of the application, different pixels in the target area may correspond to different rendering proportions. Therefore, when the method is applied to a scene of applying makeup to a facial image, natural transition exists between the makeup area and the non-makeup area, and a better makeup effect is achieved.
In one embodiment, as shown in fig. 4, the first image is a facial image, the target area is a target make-up area, and step S101 may include:
step 401: determining a sample make-up area in the sample facial image;
step 402: the sample makeup area is affine transformed to a face area in the first image according to the feature points of the first image to obtain a target area.
In the embodiment of the present application, a sample face is set, as shown in fig. 5, in the image of the sample face, there are a plurality of predetermined feature points 501, these feature points 501 form a plurality of triangular areas of the human face, and at least one sample make-up area, that is, an area of a broken line in the figure, is set on the sample face. Such as sample eye make-upareas 502 and 503, sample lip make-uparea 504, and sample cheek make-upareas 505 and 506. Wherein the sample lip make-uparea 504 coincides with the lip area. In the user face image, predetermined feature points are also present, so that the sample makeup area of the sample image can be affine transformed into the user face image.
Affine transformation, also called affine mapping, refers to the transformation of one vector space into another vector space by performing a linear transformation and a translation. Affine transformation is geometrically defined as an affine transformation between two vector spaces or affine mapping consisting of a non-singular linear transformation (transformation using a linear function) followed by a translational transformation. According to the method and the device for applying makeup to the face of the user, the sample makeup area is projected into the face image of the user through affine transformation, so that the makeup area can be determined in the face image of the user, and the makeup area accords with the face characteristics of the specific user.
According to the method and the device for determining the facial features of the user, the sample facial regions can be affine transformed into the facial images of the user, so that the target regions can be determined in the facial images of the user, and the facial regions which accord with the personal facial features of the user can be determined according to the actual facial images of the user. For example, some users have larger regions of eyes and corresponding regions of makeup on the eyes are correspondingly larger.
In one example, the target region may include a first target region and a second target region. Further, step S102 may include: determining a first target image block comprising a first target area in the first image according to the first target area; step S104 may include: in a second target area corresponding to the first target area, the content in the second target image block is fused with the content in the first target image block. For example: the first image is a facial image and the first target region is a facial organ region, such as an eye region (eye region or eyes). The second target region may be an circumscribed region of the facial organ, such as an circumscribed region of the eyes where makeup is desired.
In some cases, the second target area may be larger than the first target area. For example: the target make-up area is larger than the corresponding facial organ area, e.g., the eye make-up area is larger than the eye area. In other cases, the second target area may be smaller than the first target area. For example: the target make-up area is smaller than the corresponding facial organ area, e.g., the cheek make-up area is smaller than the cheek area.
In one embodiment, as shown in fig. 6, determining a second target image block in the second image from the first image block includes:
step 601: determining at least one image block to be selected in the second image;
step 602: and determining a second target image block from the image blocks to be selected according to the similarity between the image blocks to be selected and the first target image block.
In the embodiment of the application, the image block to be selected with the highest similarity with the first target image block can be determined from the image blocks to be selected and used as the second target image block.
In the embodiment of the application, the sliding windows with different specifications can be utilized to find the second target image block which has the most similar characteristics with the first target image block.
Specifically, the image block to be selected closest to the first target image block may be determined as the second target image block Based on a neuron Similarity (Neural Patch-Based Similarity).
According to the method and the device for determining the second target image block in the second image, the similarity is utilized, so that migrated second image content can be adapted to content in the first target image block, and the fused first image is more coordinated.
In one example of the present application, as shown in fig. 7, the image processing method includes:
step 701: a second image is obtained. The second image contains the user's cosmetic needs. The second image may in particular be a picture.
Step 702: and obtaining a dressing appearance diagram, namely a sample face image. The cosmetic case graphic includes at least one make-up area.
Step 703: and splitting the first image according to the dressing pattern diagram to obtain a plurality of first image blocks. Specifically, the first image may be split into a plurality of first image blocks according to the characteristics of the length, width, outer contour, etc. of the eyebrows. Specifically, facial organs (target areas) to be made up in the first image may be determined according to the makeup appearance map, and then the first image may be split according to the facial organs to be made up.
Step 704: a first target image block is determined in the first image block.
Step 705: the second target image block closest to the first target image block is found in the second image according to the VGG (Visual Geometry Group Network), visual geometry grid set.
Step 706: a mask layer corresponding to the target area is selected.
Step 707: and fusing the content of the second target image block into the target area by using the mask layer to realize migration of the makeup. Facial image segmentation techniques such as facial segmentation are utilized to find makeup areas such as periocular, lips, etc. corresponding to the face.
In one example of the present application, the second image may be a landscape as shown in fig. 8B. The first image may be a facial image, and the eye make-up effect is shown in fig. 8A.
In one example of the present application, the second image may be a portrait as shown in fig. 9B. The first image may be a facial image, and the face make-up effect is shown in fig. 9A.
The embodiment of the application also provides an image processing apparatus, as shown in fig. 10, including:
target area module 1001: for determining a target region in the first image;
first image object tile module 1002: the first target image block is used for determining a first image including the target area according to the target area;
the second target image block module 1003: the method comprises the steps of determining a second target image block in a second image according to a first target image block;
fusion module 1004: for fusing the content in the second target image block with the content in the first target image block in the target area.
In one embodiment, as shown in fig. 11, thefusion module 1004 includes:
first mask layer unit 1101: the target mask layer is used for determining a target mask layer corresponding to the target area from a plurality of preset mask layers, and comprises a first presentation proportion and a second presentation proportion;
the first presentation unit 1102: for presenting the corresponding content of the first target image block at a first presentation scale and the corresponding content of the second target image block at a second presentation scale in the target area.
In one embodiment, as shown in fig. 12, thefusion module 1004 includes:
a second mask layer unit 1201: the method comprises the steps of determining a target mask layer corresponding to a target area from a plurality of preset mask layers, wherein the target mask layer comprises a first presentation proportion and a second presentation proportion of each pixel in the target area;
a second presentation unit 1202: for each pixel in the target area, presenting the corresponding content of the first target image block at a first presentation scale corresponding to the pixel on the pixel, and presenting the corresponding content of the second target image block at a second presentation scale corresponding to the pixel.
In one embodiment, as shown in fig. 13, the first image is a facial image, the target area is a target make-up area, and thetarget area module 1001 includes:
make-up area determination module 1301: the method comprises the steps of determining a sample makeup area in a sample face image according to a target area;
affine transformation module 1302: the sample makeup area is affine transformed to a face area in the first image according to the feature points of the first image to obtain a target area.
In one embodiment, as shown in fig. 14, the second targetimage block module 1003 includes:
a to-be-selected image block unit 1501: for determining at least one image block to be selected in the second image;
a second target image block unit 1502: and determining a second target image block from the image blocks to be selected according to the similarity between the image blocks to be selected and the first target image block.
The functions of each module in each apparatus of the embodiments of the present application may be referred to the corresponding descriptions in the above methods, which are not described herein again.
According to embodiments of the present application, an electronic device and a readable storage medium are also provided.
As shown in fig. 15, there is a block diagram of an electronic device according to a method of image processing according to an embodiment of the present application. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the application described and/or claimed herein.
As shown in fig. 15, the electronic device includes: one or more processors 1601,memory 1602, and interfaces for connecting the components, including a high-speed interface and a low-speed interface. The various components are interconnected using different buses and may be mounted on a common motherboard or in other manners as desired. The processor may process instructions executing within the electronic device, including instructions stored in or on memory to display graphical information of the GUI on an external input/output device, such as a display device coupled to the interface. In other embodiments, multiple processors and/or multiple buses may be used, if desired, along with multiple memories and multiple memories. Also, multiple electronic devices may be connected, each providing a portion of the necessary operations (e.g., as a server array, a set of blade servers, or a multiprocessor system). One processor 1601 is shown in fig. 15 as an example.
Memory 1602 is a non-transitory computer-readable storage medium provided herein. Wherein the memory stores instructions executable by the at least one processor to cause the at least one processor to perform the methods of image processing provided herein. The non-transitory computer readable storage medium of the present application stores computer instructions for causing a computer to perform the method of image processing provided herein.
Thememory 1602 is a non-transitory computer readable storage medium that can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as program instructions/modules (e.g., thetarget region module 1001, the first targetimage block module 1002, the second targetimage block module 1003, and thefusion module 1004 shown in fig. 10) corresponding to the method of image processing in the embodiments of the present application. The processor 1601 executes various functional applications of the server and data processing, i.e., a method of implementing image processing in the above-described method embodiment, by executing non-transitory software programs, instructions, and modules stored in thememory 1602.
Memory 1602 may include a storage program area that may store an operating system, at least one application program required for functionality, and a storage data area; the storage data area may store data created according to the use of the electronic device for image processing, or the like. In addition,memory 1602 may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments,memory 1602 may optionally include memory located remotely from processor 1601, which may be connected to the image processing electronics by a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The electronic device of the method of image processing may further include: aninput device 1603 and anoutput device 1604. The processor 1601,memory 1602,input device 1603, andoutput device 1604 may be connected by a bus or otherwise, for example in fig. 15.
Theinput device 1603 may receive input numeric or character information and generate key signal inputs related to user settings and function control of the image processing electronic device, such as a touch screen, keypad, mouse, trackpad, touchpad, pointer stick, one or more mouse buttons, trackball, joystick, and like input devices. Theoutput devices 1604 may include a display device, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors), among others. The display device may include, but is not limited to, a Liquid Crystal Display (LCD), a Light Emitting Diode (LED) display, and a plasma display. In some implementations, the display device may be a touch screen.
Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, application specific ASIC (application specific integrated circuit), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.
These computing programs (also referred to as programs, software applications, or code) include machine instructions for a programmable processor, and may be implemented in a high-level procedural and/or object-oriented programming language, and/or in assembly/machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and/or device (e.g., magnetic discs, optical disks, memory, programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and/or data to a programmable processor.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), and the internet.
The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
According to the technical scheme of the embodiment of the application, the target area in the first image is determined, the first target image block is determined, the second target image block in the second image with certain correlation with the first target image block is determined, and finally the content in the second target image block is fused to the target area in the first image, so that the technical problem that the image shooting mode seeks diversified roads is solved, and the technical effect of enriching the image shooting means is achieved. It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps described in the present application may be performed in parallel, sequentially, or in a different order, provided that the desired results of the technical solutions disclosed in the present application can be achieved, and are not limited herein.
The above embodiments do not limit the scope of the application. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application are intended to be included within the scope of the present application.