Detailed Description
Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is only exemplary and is not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. In addition, in the following description, descriptions of well-known structures and techniques are omitted so as not to unnecessarily obscure the concepts of the present disclosure.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "comprises," "comprising," and/or the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.
All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It should be noted that the terms used herein should be construed to have meanings consistent with the context of the present specification and should not be construed in an idealized or overly formal manner.
Where a convention analogous to "at least one of A, B and C, etc." is used, in general such a convention should be interpreted in accordance with the meaning of one of skill in the art having generally understood the convention (e.g., "a system having at least one of A, B and C" would include, but not be limited to, systems having a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.). Where a formulation similar to at least one of "A, B or C, etc." is used, in general such a formulation should be interpreted in accordance with the ordinary understanding of one skilled in the art (e.g. "a system with at least one of A, B or C" would include but not be limited to systems with a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.).
Image deblurring refers to repairing a blurred image that has been generated by some algorithm, and ultimately improving the visual effect of the image. Aiming at the application occasion of a given image, the integral and local characteristics of the image are purposefully emphasized, the original unclear image is made clear or certain business characteristics are emphasized, the difference between different object characteristics in the image is enlarged, the non-business characteristics are restrained, the image quality is improved, the information quantity is enriched, and the image interpretation and recognition effects are enhanced.
Currently, in the process of asking a question by a user, it is often required to shoot a physical object and send the shot image to a client so that the client can answer according to the image. In the shooting process, the problems of more noise, lower resolution and the like of the shot image can occur due to factors such as inaccurate focusing, aberration of an optical system and the like, so that the system can not accurately recognize the image. In addition, the influence of external illumination, environmental noise and the like can cause the problems of more noise, lower resolution and the like of the shot image. In this case, the user is often required to take a plurality of shots again, which not only increases the ineffective communication time, but also reduces the communication efficiency.
In view of the above, aiming at the condition of blurring of photographed images, the method and the device can combine the generated countermeasure network, perform deblurring treatment on images uploaded by users through the image deblurring model which is optimized through continuous learning in the earlier stage, generate images with high image resolution and high system recognition rate, enhance the actual use experience of the users, avoid repeated operation of the users, increase the accuracy of system recognition problems, and further improve the resolution of problems encountered in the use process of the users.
Specifically, the embodiment of the disclosure provides an image processing method, an image processing device, an electronic device, a readable storage medium and a computer program product, which can effectively reduce noise and interference in an image to be identified, effectively solve the problem of image blurring and improve the success rate of the image to be identified. The image processing method comprises the steps of responding to a service processing request, obtaining an image to be processed carried in the service processing request, segmenting the image to be processed to obtain multiple frames of sub-images, respectively processing image data of the multiple frames of sub-images by using a feature extraction model to determine a first target sub-image from the multiple frames of sub-images, and processing the image data of the first target sub-image by using an image processing model to obtain a target image.
It should be noted that the image processing method, apparatus, electronic device, storage medium, and program product determined in the embodiments of the present disclosure may be used in the field of artificial intelligence technology and the field of finance, and may also be used in any field other than the field of artificial intelligence technology and the field of finance, and the specific application field thereof is not limited.
In the technical scheme of the disclosure, the related processes of collecting, storing, using, processing, transmitting, providing, disclosing, applying and the like of the personal information of the user all conform to the regulations of related laws and regulations, necessary security measures are adopted, and the public order harmony is not violated.
In the technical scheme of the disclosure, the authorization or consent of the user is obtained before the personal information of the user is obtained or acquired.
Fig. 1 schematically illustrates an exemplary system architecture to which image processing methods and apparatuses may be applied according to embodiments of the present disclosure, and it should be noted that fig. 1 illustrates only an example of a system architecture to which embodiments of the present disclosure may be applied, so as to assist those skilled in the art in understanding the technical content of the present disclosure, but not to mean that embodiments of the present disclosure may not be applied to other devices, systems, environments, or scenes.
As shown in fig. 1, a system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used as a medium to provide communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and/or wireless communication links, and the like.
The user may interact with the server 105 via the network 104 using the terminal devices 101, 102, 103 to receive or send messages or the like. Various communication client applications may be installed on the terminal devices 101, 102, 103, such as financial class applications, shopping class applications, web browser applications, search class applications, instant messaging tools, mailbox clients and/or social platform software, to name a few.
The terminal devices 101, 102, 103 may be various electronic devices with cameras and/or display screens, including but not limited to smartphones, tablets, laptop and desktop computers, and the like.
The server 105 may be a server providing various services, for example, a background management server (merely an example) providing support for a service processing request carrying an image to be processed transmitted by a user using the terminal devices 101, 102, 103. The background management server can analyze and process the received data such as the service processing request, and feed back the processing result (for example, the background management server can firstly perform denoising and defuzzification processing on the image to be processed in the service processing request, so as to obtain a clear image) to the terminal equipment.
It should be noted that, the image processing method provided by the embodiment of the present disclosure may be generally executed by the server 105. Accordingly, the image processing apparatus provided by the embodiments of the present disclosure may be generally provided in the server 105. The image processing method provided by the embodiments of the present disclosure may also be performed by a server or a server cluster that is different from the server 105 and is capable of communicating with the terminal devices 101, 102, 103 and/or the server 105. Accordingly, the image processing apparatus provided by the embodiments of the present disclosure may also be provided in a server or a server cluster that is different from the server 105 and is capable of communicating with the terminal devices 101, 102, 103 and/or the server 105. Or the image processing method provided by the embodiment of the present disclosure may be performed by the terminal apparatus 101, 102, or 103, or may be performed by another terminal apparatus other than the terminal apparatus 101, 102, or 103. Accordingly, the image processing apparatus provided by the embodiments of the present disclosure may also be provided in the terminal device 101, 102, or 103, or in another terminal device different from the terminal device 101, 102, or 103.
It should be understood that the number of terminal devices, networks and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Fig. 2 schematically shows a flowchart of an image processing method according to an embodiment of the present disclosure.
As shown in FIG. 2, the method includes operations S201-S204.
In operation S201, in response to the service processing request, a to-be-processed image carried in the service processing request is acquired.
In operation S202, the image to be processed is segmented, and a multi-frame sub-image is obtained.
In operation S203, image data of the plurality of sub-images are respectively processed using the feature extraction model to determine a first target sub-image from among the plurality of sub-images.
In operation S204, image data of the first target sub-image is processed using the image processing model to obtain a target image.
According to an embodiment of the present disclosure, the image to be processed may be an image associated with a service characterized by the service processing request. The image to be processed may include an object to be identified related to the business, where the object to be identified may be, for example, a bank card, a deposit card, various business documents, and the like. The identification of the object to be identified can be to identify information such as characters, patterns, colors and the like in the image to be processed.
The manner in which the image to be processed is segmented according to the embodiments of the present disclosure is not limited herein. For example, the image to be processed may be split into sub-images of the same multi-frame size. Specifically, the image to be processed may be uniformly divided into m×n rectangular blocks, M, N are integers greater than or equal to 0, and each rectangular block is characterized as a sub-image of one frame. For another example, the type of the object to be identified in the image to be processed may be determined first, and the segmentation may be performed according to a region segmentation rule corresponding to the type. Specifically, when the image to be processed is an image including a bank card, a region segmentation rule corresponding to the type of the bank card may be used to segment a key region of the bank card, for example, regions such as a bank card title line, a bank card number, a bank card mark, and a background may be segmented respectively. The obtained multi-frame sub-images can be sub-images of areas including a bank card title line, a bank card number, a bank card mark, a background and the like.
According to an embodiment of the present disclosure, the feature extraction model may output feature vectors from input image data. The feature extraction model may be trained based on any existing feature extraction network architecture. Existing feature extraction network architectures may include, but are not limited to MobileNetV2, shuffleNetV2, PVANet, etc.
According to embodiments of the present disclosure, an image processing model may be used to generate a target image from an input first target sub-image. The image processing model may be any type of image generation model, such as pixelRNN/CNN, GAN (generated against network model), etc.
According to the embodiment of the disclosure, a multi-frame sub-image is obtained by segmenting an image to be processed, a first target sub-image is determined from the multi-frame sub-image by using a feature extraction model, and then the first target sub-image is processed by a graphic processing model to obtain the target image. Before generating the target image, the technical means of dividing the image into multiple frames of sub-images and determining the target image from the multiple frames of sub-images is adopted, so that the influence of characteristics irrelevant to the target image and noise on the generation process of the target image is reduced, the resolution of the target image is effectively improved, and the technical problem that the image to be identified in a business system cannot be accurately identified due to more noise and lower resolution is at least partially overcome.
According to an embodiment of the present disclosure, the service type may also be carried in the service processing request.
According to embodiments of the present disclosure, the service type may include, for example, transfer, deposit, loan, charge settlement, service agency, and the like.
According to an embodiment of the present disclosure, operation S203 may include determining a reference feature vector based on a service type, wherein the reference feature vector includes image data of a reference image associated with the service type processed using a feature extraction model, processing image data of a plurality of sub-images using the feature extraction model to obtain a plurality of sub-feature vectors, respectively, determining a similarity of the sub-feature vector to the reference feature vector for each of the plurality of sub-feature vectors to obtain a plurality of similarities, and determining a first target sub-image from the plurality of sub-images based on the plurality of similarities.
According to an embodiment of the present disclosure, the reference image may be a standard image associated with a service type indicated by the service processing request. For example, in processing a transfer transaction, the identification of the bank card number is required, and the corresponding reference image may be an image including the bank card number and having a high identification accuracy.
According to embodiments of the present disclosure, the similarity may be obtained by calculating distances of the sub-feature vector and the reference feature vector in the vector space. Specifically, the closer the distance between the sub-feature vector and the reference feature vector in the vector space, the higher the similarity between the sub-feature vector and the reference feature vector, and the further the distance between the sub-feature vector and the reference feature vector in the vector space, the lower the similarity between the sub-feature vector and the reference feature vector. The calculation of the distance of the feature vector in the vector space may be implemented by a euclidean distance algorithm, a manhattan distance algorithm, a mahalanobis distance algorithm, or the like, which is not limited herein.
According to the embodiment of the disclosure, the similarity can also be calculated by adopting a pearson correlation coefficient algorithm, a cosine similarity algorithm and the like.
According to an embodiment of the disclosure, the first target sub-image may be a sub-image closest to the reference image in the multiple frames of sub-images, that is, the sub-image corresponding to the maximum similarity among the calculated multiple similarities may be determined to be the first target sub-image.
According to the embodiment of the disclosure, through segmentation and similarity calculation, the condition that missing image information may exist in the determined first target sub-image or more useless image information is contained, so that in some embodiments, before the generation of the target image, the method can further comprise the following operation of intercepting the second target sub-image from the image to be processed based on the first target sub-image. Wherein operation S204 may include an operation of processing image data of the second target sub-image using the image processing model to obtain the target image.
According to an embodiment of the disclosure, capturing a second target sub-image from an image to be processed based on a first target sub-image may include determining a position of the first target sub-image in the image to be processed to obtain position information, and capturing the second target sub-image from the image to be processed based on the position information.
According to the embodiment of the disclosure, the position information of the second target sub-image can be obtained by proportionally expanding or shrinking the area around the first target sub-image according to the position information of the first target sub-image, and then the second target sub-image can be obtained by intercepting. The settings that need to be enlarged or reduced and the proportions can be adapted according to the actual needs.
According to an embodiment of the present disclosure, the size of the second target sub-image is larger than the size of the first target sub-image.
According to the embodiment of the disclosure, particularly, when the image to be processed is a bank card, the first target sub-image is a part of the Unionpay card number, the second target sub-image can be all the Unionpay card numbers, and when the first target sub-image is a part of the Unionpay mark, the second target sub-image can be all the Unionpay marks.
Alternatively, the size of the second target sub-image may be smaller than the size of the first target sub-image according to embodiments of the present disclosure. For example, when the image to be processed is a bank card, the first target sub-image may be all the Unionpay card numbers from which other invalid information is removed when the first target sub-image is other invalid information than all the Unionpay card numbers, and the second target sub-image may be all the Unionpay marks when the first target sub-image is other invalid information than all the Unionpay marks.
According to an embodiment of the present disclosure, operation S204 may further include the following operations:
And processing the image data of the second target sub-image by using the image processing model to obtain a target image. I.e. the target image is generated using the re-determined sub-image, thereby further increasing the identifiable probability of the generated image.
According to embodiments of the present disclosure, the generated target image may be made available for invocation by other systems in the business system. For example, the other system is an intelligent customer service answering system of a bank, and the system can acquire information required by a service by processing a target image. Specifically, the image data of the target image may be processed using the image recognition model, resulting in an image recognition result of the image to be processed.
Fig. 3 schematically illustrates a flowchart of an image processing model training method according to an embodiment of the present disclosure.
As shown in FIG. 3, the method includes operations S301-S305.
In operation S301, a training sample set is acquired, wherein the training sample set includes a plurality of sample images.
In operation S302, for each of a plurality of sample images, the sample image is segmented to obtain a plurality of sub-sample images.
In operation S303, image data of a plurality of sub-sample images are respectively processed using a feature extraction model to determine a first sub-sample image from the plurality of sub-sample images.
In operation S304, a second sub-sample image is generated based on the first sub-sample image.
In operation S305, an initial image processing model is trained using image data of a first sub-sample image and image data of a second sub-sample image, resulting in an image processing model.
According to the embodiment of the disclosure, the training sample set can be selected according to the actual scene of the banking and finance industry inquiry customer service, and the actual scene can comprise reading of a card number of a bank card, shooting of photos of the front side and the back side of an identity card, shooting of problems encountered by other mobile phones when the APP is used, and the like. The sample image may be an image associated with a service characterized by the service processing request. The sample image may include an object to be identified related to the business, where the object to be identified may be, for example, a bank card, a deposit card, various business documents, and the like. The object to be identified may be identified by identifying information such as characters, patterns, colors, etc. in the sample image.
The manner in which the sample image is segmented according to embodiments of the present disclosure is not limited herein. For example, the sample image may be split into sub-images of the same multi-frame size. Specifically, the sample image may be uniformly divided into m×n rectangular blocks, M, N being integers greater than or equal to 0, and each rectangular block is characterized as a sub-image of one frame. For another example, the type of the object to be identified in the sample image may be determined first, and the segmentation may be performed according to a region segmentation rule corresponding to the type. Specifically, when the sample image is an image including a bank card, a region segmentation rule corresponding to the type of the bank card may be used to segment a key region of the bank card, for example, regions such as a bank card title line, a bank card number, a bank card flag, and a background may be segmented respectively. The obtained multi-frame sub-images can be sub-images of areas including a bank card title line, a bank card number, a bank card mark, a background and the like.
According to an embodiment of the present disclosure, the feature extraction model may output feature vectors from input image data. In another embodiment, the feature extraction network may further obtain a final output after adding the sample image by elements, and the output result may be a result of determining whether the sample image is a real image. The feature extraction model can be a lightweight feature extraction network model or can be trained based on any existing feature extraction network architecture. Existing feature extraction network architectures may include, but are not limited to MobileNetV2, shuffleNetV2, PVANet, etc.
According to the method and the device for processing the multi-frame sub-sample images, the first sub-sample image is determined from the multi-frame sub-sample images by using the feature extraction model, the multi-frame sub-sample images in the training sample set can be subjected to feature extraction to obtain the first sub-sample image, parameters of the image processing model can be reduced, training time of the image processing model can be shortened, and efficiency of training the image processing model is improved.
According to an embodiment of the present disclosure, operation S304 may further include an operation of generating a second sub-sample image based on the first sub-sample image, including performing a blurring process on the first sub-sample image to obtain the second sub-sample image, wherein the blurring process includes a gaussian blurring process and/or a motion blurring process.
According to an embodiment of the present disclosure, the first sub-sample image may be a clear image, and the second sub-sample image may be a blurred image corresponding to the clear image. Specifically, a first sub-sample image and a second sub-sample image can be obtained by using a high-speed camera to shoot a video of a target object in a moving state, then performing frame cropping on the video, and finding out a clear image and a blurred image corresponding to the clear image. The blurring process image may be a process of generating a blurred image in a simulated manner according to a cause of generation of the blurred image in an actual scene. For example, it may be a gaussian blur process of different intensities and/or a motion blur process of different intensities. Gaussian blur and motion blur are two main types of causes that lead to blurring of a captured picture. Specifically, a blurred image of the first sub-sample image may be generated as the second sub-sample image using a gaussian blur kernel when using a gaussian blur process. The larger the convolution kernel is, the more obvious the blurring effect is on the first sub-sample image. Preferably, since the sample images are mostly clear images, the albumentations library may be used to blur the first sub-sample image, and a3×3 convolution kernel may be used for the processing.
According to embodiments of the present disclosure, the initial image processing model may employ a model architecture of a generative countermeasure module. The training method of the image processing model will be further described below by taking the initial image processing model as a generating type countermeasure module as an example.
According to embodiments of the present disclosure, an initial image processing model may be composed of a generator and a arbiter.
According to an embodiment of the present disclosure, operation S305 may further include processing the image data of the second sub-sample image using the generator to obtain the image data of the first generated image, processing the image data of the first sub-sample image and the image data of the first generated image using the arbiter to obtain the discrimination data, and training the generator and the arbiter using the discrimination data to obtain the image processing model.
According to an embodiment of the present disclosure, the initial image processing model may be a generative antagonistic neural network model including a generator and a arbiter. The generated antagonistic neural network model can be trained by the first sub-sample image and the second sub-sample image. Specifically, the generator and the arbiter may be trained based on discrimination data obtained by the arbiter, and the generator loss function and the arbiter loss function may be used to determine the loss of the generated type antagonistic neural network model, respectively, and the training of the generated type antagonistic neural network model may be completed by minimizing the model loss.
According to embodiments of the present disclosure, the arbiter (D) may be used to determine whether the generator (G) generates a dummy picture or a real picture, i.e. whether the picture is true or false. The loss function of the discriminator (D) may be as shown in equation (1).
lossD＝Dfake-Dreal (1)
Where lossD may represent the loss of the arbiter (D), Dfake may represent the probability of a determination being a dummy picture, and Dreal may represent the probability of a determination being a true picture. Therefore, when Dfake is smaller and Dreal is larger, it can be stated that the discriminator (D) can more accurately determine the authenticity of the picture, and therefore the smaller the loss function loss_d, the better.
According to an embodiment of the present disclosure, a loss function of the generator (G) for making a difference between the generated image and the clear image smaller and smaller may be as shown in formula (2).
lossG=content_loss+t×adv_loss (2)
The lossG may represent a loss of the generator (G), the content_loss may represent that the difference between the generated image and the clear image is smaller and smaller, and the t may represent a super parameter set according to the need, so that the discriminator determines the generated image as a real image, and finally, an image deblurring effect is achieved.
In accordance with embodiments of the present disclosure, in the generative antagonistic neural network model, the feature extraction network architecture input to the arbiter may be replaced with a lightweight feature extraction network, for example, using mobileNet feature extraction networks to extract the image data of the first sub-sample image and the image data features of the first generated image. By using the lightweight feature extraction network, a large number of training sample sets are not needed when the image processing model is trained, and parameters of the image processing model can be reduced, so that the training time of the image processing model is shortened, and the image processing model is lightened.
According to the embodiment of the disclosure, the discriminators comprise a global discriminator and a local discriminator, the discrimination data comprise first discrimination data and second discrimination data, the image data of the first sub-sample image and the image data of the first generated image are processed by the discriminators to obtain discrimination data, the image data of the first sub-sample image and the image data of the first generated image are processed by the global discriminator to obtain first discrimination data, the first sub-sample image and the first generated image are respectively processed based on a prediction local clipping strategy to obtain third sub-sample image and second generated image, and the image data of the third sub-sample image and the image data of the second generated image are processed by the local discriminator to obtain second discrimination data.
According to an embodiment of the present disclosure, adv_loss in equation (2) may be as shown in equation (3).
adv_loss=-Dlocal(fake)-0.5×Dglobal(fake) (3)
Where Dlocal(fake) may represent the probability that the local arbiter determines that the pseudo-picture is a pseudo-picture, and Dglobal(fake) may represent the probability that the global arbiter determines that the pseudo-picture is a pseudo-picture. For adv_loss, since the purpose of the generator is to generate a pseudo picture closer to a real picture, so that the arbiter can judge the pseudo picture as the real picture, the smaller the adv_loss is, the smaller the loss is, the finer texture generation can be ensured in the generated picture, and therefore the picture deblurring effect is improved.
Taking as an example the generated-type antagonistic neural network model employed by the initial image model, the generated-type antagonistic neural network model may include a generator and a arbiter, according to an embodiment of the present disclosure. The generator wishes to generate an image closer to the real picture by successive iterations so that the discriminator determines the pseudo-picture generated by the generator as the real picture, the discriminator being the pseudo-picture that it wishes to be able to discriminate the generator. Therefore, after a large number of iterations, the pseudo image generated by the generator is more and more similar to the real picture, and the discriminator cannot identify the authenticity of the generated pseudo picture.
According to an embodiment of the present disclosure, the first generated image may be a clear image generated by the generator from the blurred second sub-sample image, which is different from the first sub-sample image. Discrimination data can be obtained by comparing the image data of the first sub-sample image with the image data of the first generated image by the discriminator. And operating the training generator and the discriminator by a large number of iterative computations, model parameter adjustment and the like according to the discrimination data to obtain a trained image processing model.
According to the embodiment of the disclosure, the discriminators in the generated antagonistic neural network model can be divided into a global discriminator and a local discriminator, one image in the training sample set can be input into the global discriminator as a whole image, the local image after the image is subjected to local random clipping can be input into the local discriminator, and then the probability that the discriminators are true pictures for the pictures generated by the generator can be output after a series of convolution operations.
According to an embodiment of the present disclosure, the first sub-sample image may be input as an entire image into the global arbiter, and accordingly, the first generated image may also be an entire image.
According to embodiments of the present disclosure, the predictive local clipping policy may be clipping according to the same principle as the region. The specific clipping method can be random clipping, regional clipping and the like. For example, the random cropping may be to randomly crop the first sub-sample image to obtain a third sub-sample image and randomly crop the first generated image to obtain a second generated image. The region clipping may be clipping according to a key region, for example, in the case where the first sub-sample image and the first generated image respectively include a bank card, the third sub-sample image and the second generated image may be clipping regions such as a bank card title line, a bank card number, a bank card flag, a background, and the like. The third sub-sample image may be input to the local arbiter as a cropped local image, and the second generated image may be a cropped local image.
According to embodiments of the present disclosure, a generator loss function and a discriminant loss function may be employed to determine the loss of an initial image processing model, which is trained by minimizing model loss.
In accordance with embodiments of the present disclosure, during the training of the image processing model, it may also be selected to divide the existing training sample set into two parts, a training set and a testing set. The image processing model can be trained through a training set, and the discrimination capability of the model to the image to be processed can be evaluated through a testing set. Therefore, the problem that the image processing model takes the characteristics of the training sample as the common characteristics of the potential sample in the training process, so that the training sample is better in the learning and training process, and the fitting is caused can be avoided.
According to embodiments of the present disclosure, a training sample set may be partitioned using a cross-validation method. Specifically, the training sample set may be divided into k parts, where k-1 parts may be used as the training set, and the remaining 1 part may be used as the test set, so that k sets of training and testing are performed according to the divided k-1 sets of training sets and 1 set of test sets, and finally an average value of k test results may be returned. The k value can be selected to be adaptively adjusted according to actual needs or experience. Preferably, k is 10 or more, so as to ensure the stability of the output result.
According to embodiments of the present disclosure, the training process for the image processing model may further include creating a virtual environment and installing relevant dependencies in the virtual environment. Specifically, a virtual environment of python3.9 may be created first, and then the modules pytorch1.0.1, opencv, numpy, etc. are installed in the environment.
According to embodiments of the present disclosure, the training process for the image processing model may further include modifying the configuration file. Specifically, the initialization setting may be performed on a plurality of training parameters under a config file in the image processing model, for example, information such as a training sample set path, an epoch number, a batch_size, a learning rate, and the like.
According to embodiments of the present disclosure, the training process for the image processing model may also be visualized. Specifically, tensorboard visualization tools can be adopted to display tensors, index changes of the network graph and the like in the model training process, and a storage path of a file and a loading path of a training sample set used can be set when the model training system is used.
According to the embodiment of the disclosure, in the process of training the model, whether optimization is needed or not can be judged through fitting conditions in the training process. Specifically, the judgment can be made by the change condition of the loss function of the producer and/or the arbiter when the model is trained. If the loss function oscillates, the whole model can be indicated to be unstable. The accuracy of the training set and the testing set in the training process can be judged, and if the accuracy of the training set is found to be higher, the accuracy of the testing set is relatively lower, the situation that the image processing model is over-fitted can be indicated, and the model needs to be optimized.
According to embodiments of the present disclosure, the image processing model may also be optimized using data enhancement techniques. Specifically, a data enhancement technique may be used on the training set, for example, data is added through random clipping, color dithering, scale variation, contrast transformation, and the like, so that not only can the generalization capability of the model be improved, but also noise data can be added, and the robustness of the model is enhanced.
According to embodiments of the present disclosure, the model may also be optimized by modifying the learning rate. By using the varying learning rate, the fitting capacity of the image processing model can be increased, thereby improving the accuracy of the model. Specifically, a larger learning rate can be used for training first, a loss function curve and a convergence curve of the test set accuracy are observed, when the falling speed of the loss function curve and the rising speed of the convergence curve of the test set accuracy are reduced, the learning rate can be reduced, and the learning rate can be repeatedly reduced for a plurality of times until the falling speed of the loss function curve and the rising speed of the convergence curve of the test set accuracy are not affected.
The image processing model may also be optimized using a deepened network layer number according to embodiments of the present disclosure. Specifically, the number of layers of the neural network can be properly deepened, the turning point from the rising to the falling of the accuracy in the test set training process can be found, and the performance of the model is optimized by deepening the depth of the number of layers of the network.
According to embodiments of the present disclosure, the trained image processing model may also be validated. An appropriate model may be selected first. Specifically, after training the training sample set using a GPU (graphics processor), a plurality of available model parameter files may be generated, and verification may be performed by determining and selecting an appropriate model.
According to the embodiment of the disclosure, after a model is selected, an original test image to be deblurred may be input into the model one by one, and the model may output a plurality of deblurred clear images corresponding to the test image to be deblurred and stored in a folder.
According to the embodiments of the present disclosure, the deblurring effect can be viewed after outputting a clear image. Specifically, the output clear image can be stored in a folder, the comparison effect of the original test image and the defuzzified clear image can be checked in the folder, and the verification of the trained image processing model can be realized.
According to the embodiment of the disclosure, an initial sample image is acquired, wherein an initial sample set comprises a plurality of initial sample images, data enhancement processing is carried out on each initial sample image in the plurality of initial sample images to obtain a plurality of sample images, and the data enhancement processing comprises any one or more of image clipping, image rotation, scale change, color dithering, contrast conversion and noise addition.
According to embodiments of the present disclosure, the sample image may be an initial sample image after data enhancement. Data enhancement may include any one or more of image cropping, image rotation, scale change, color dithering, contrast conversion, and noise addition. For example, the image rotation may be that the initial sample image is rotated by taking the upper left corner of the image as a rotation center to obtain a sample image, the random rotation angle may be-45 degrees to 45 degrees, the scale change may be that the initial image is randomly scaled to obtain a sample image, the scaling range may be 0.5-1.5 times of the size of the initial image, the noise addition may be that the salt and pepper noise is increased to obtain a sample image, and the salt and pepper noise addition is realized by changing some pixel values of the initial image into 0 or 255 randomly.
According to the embodiment of the disclosure, the data enhancement operation may be performed on the first sub-sample image before the blurring process is performed on the first sub-sample image, for example, the rotation, scaling, cutting and other operations may be performed on the first sub-sample image, and then the blurring process is performed on the data-enhanced first sub-sample image to generate the second sub-sample image.
According to the embodiment of the disclosure, the data enhancement is performed on the basis of the initial sample image, so that a model trained by the training sample set after the data enhancement can be used under the condition that other training conditions are unchanged, the defuzzification effect is improved, and the image with higher resolution is generated.
It should be noted that, unless there is an execution sequence between different operations or an execution sequence between different operations in technical implementation, the execution sequence between multiple operations may be different, and multiple operations may also be executed simultaneously in the embodiment of the disclosure.
Fig. 4 schematically shows a block diagram of an image processing apparatus according to an embodiment of the present disclosure.
As shown in fig. 4, the image processing apparatus 400 includes a response module 410, a first segmentation module 420, a first processing module 430, and a second processing module 440.
The response module 410 is configured to respond to a service processing request, and obtain an image to be processed carried in the service processing request.
The first segmentation module 420 is configured to segment the image to be processed to obtain multiple sub-images.
The first processing module 430 is configured to process the image data of the plurality of frames of the sub-images, respectively, using the feature extraction model to determine a first target sub-image from the plurality of frames of the sub-images.
The second processing module 440 is configured to process the image data of the first target sub-image using an image processing model to obtain a target image.
According to an embodiment of the present disclosure, the first processing module 430 further includes a first determining unit, a first processing unit, a second determining unit, and a third determining unit.
The first determining unit is configured to determine a reference feature vector based on the service type, wherein the reference feature vector is obtained by processing image data of a reference image associated with the service type using the feature extraction model.
The first processing unit is used for respectively processing the image data of the multiple frames of the sub-images by using the characteristic extraction model to obtain multiple sub-characteristic vectors.
The second determining unit is configured to determine, for each of the plurality of sub-feature vectors, a similarity between the sub-feature vector and the reference feature vector, and obtain a plurality of similarities.
The third determining unit is configured to determine the first target sub-image from among the sub-images of a plurality of frames based on the plurality of similarities.
According to an embodiment of the present disclosure, the image processing apparatus 400 further includes an interception module.
The intercepting module is used for intercepting and obtaining a second target sub-image from the image to be processed based on the first target sub-image.
According to an embodiment of the present disclosure, the second processing unit 440 includes a second processing unit.
And the second processing unit is used for processing the image data of the second target sub-image by using the image processing model to obtain the target image.
According to an embodiment of the disclosure, the interception module comprises a fourth determination unit and an interception unit.
And the fourth determining unit is used for determining the position of the first target sub-image in the image to be processed to obtain position information.
And the intercepting unit is used for intercepting the second target sub-image from the image to be processed based on the position information.
According to an embodiment of the present disclosure, the image processing apparatus further includes a third processing unit.
And the third processing unit is used for processing the image data of the target image by using an image recognition model to obtain an image recognition result of the image to be processed. The image processing device according to the embodiment of the disclosure further comprises a first acquisition module, a second segmentation module, a third processing module, a generation module and a training module.
The first acquisition module is used for acquiring a training sample set, wherein the training sample set comprises a plurality of sample images.
The second segmentation module is used for segmenting each sample image in the plurality of sample images to obtain multi-frame sub-sample images.
The third processing module is used for respectively processing the image data of the multiple frames of the sub-sample images by using the characteristic extraction model so as to determine a first sub-sample image from the multiple frames of the sub-sample images.
The generation module is used for generating a second sub-sample image based on the first sub-sample image.
The training module is used for training an initial image processing model by using the image data of the first sub-sample image and the image data of the second sub-sample image to obtain an image processing model.
According to an embodiment of the disclosure, the generation module comprises a fourth processing unit.
And the fourth processing unit is used for carrying out blurring processing on the first sub-sample image to obtain the second sub-sample image.
According to an embodiment of the present disclosure, the training module includes a fifth processing unit, a sixth processing unit, and a training unit.
And the fifth processing unit is used for processing the image data of the second subsampled image by using the generator to obtain the image data of the first generated image.
The sixth processing unit is used for processing the image data of the first sub-sample image and the image data of the first generated image by using the discriminator to obtain discrimination data.
The training unit is used for training the generator and the discriminator by using the discrimination data to obtain the image processing model.
According to an embodiment of the present disclosure, the sixth processing unit further comprises a first processing subunit, a second processing subunit, and a third processing subunit.
The first processing subunit is configured to process the image data of the first sub-sample image and the image data of the first generated image by using the global arbiter, so as to obtain the first discrimination data.
The second processing subunit is used for respectively processing the first sub-sample image and the first generated image based on a prediction local clipping strategy to obtain a third sub-sample image and a second generated image.
And the third processing subunit is used for processing the image data of the third subsampled image and the image data of the second generated image by using the local discriminator to obtain the second discrimination data.
According to an embodiment of the present disclosure, the image processing apparatus further includes a second acquisition module, a fourth processing module.
The second acquisition module is used for acquiring initial sample images, wherein the initial sample set comprises a plurality of initial sample images.
And the fourth processing module is used for carrying out data enhancement processing on each initial sample image in the plurality of initial sample images to obtain a plurality of sample images.
Any number of modules, sub-modules, units, sub-units, or at least some of the functionality of any number of the sub-units according to embodiments of the present disclosure may be implemented in one module. Any one or more of the modules, sub-modules, units, sub-units according to embodiments of the present disclosure may be implemented as split into multiple modules. Any one or more of the modules, sub-modules, units, sub-units according to embodiments of the present disclosure may be implemented at least in part as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system-on-chip, a system-on-substrate, a system-on-package, an Application Specific Integrated Circuit (ASIC), or in any other reasonable manner of hardware or firmware that integrates or encapsulates the circuit, or in any one of or a suitable combination of three of software, hardware, and firmware. Or one or more of the modules, sub-modules, units, sub-units according to embodiments of the present disclosure may be at least partially implemented as computer program modules, which, when executed, may perform the corresponding functions.
For example, any of the response module 410, the first dividing module 420, the first processing module 430, and the second processing module 440 may be combined in one module/unit/sub-unit or any of the modules/units/sub-units may be split into a plurality of modules/units/sub-units. Or at least some of the functionality of one or more of these modules/units/sub-units may be combined with at least some of the functionality of other modules/units/sub-units and implemented in one module/unit/sub-unit. According to embodiments of the present disclosure, at least one of the response module 410, the first splitting module 420, the first processing module 430, and the second processing module 440 may be implemented at least in part as hardware circuitry, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented in hardware or firmware in any other reasonable manner of integrating or packaging the circuitry, or in any one of or a suitable combination of any of the three. Or at least one of the response module 410, the first segmentation module 420, the first processing module 430, and the second processing module 440 may be at least partially implemented as computer program modules that, when executed, perform the corresponding functions.
It should be noted that, in the embodiment of the present disclosure, the image processing apparatus portion corresponds to the image processing method portion in the embodiment of the present disclosure, and the description of the image processing apparatus portion specifically refers to the image processing method portion and is not described herein.
Fig. 5 schematically shows a block diagram of an electronic device adapted to implement an image processing method according to an embodiment of the disclosure. The electronic device shown in fig. 5 is merely an example and should not be construed to limit the functionality and scope of use of the disclosed embodiments.
As shown in fig. 5, a computer electronic device 500 according to an embodiment of the present disclosure includes a processor 501 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 502 or a program loaded from a storage section 508 into a Random Access Memory (RAM) 503. The processor 501 may include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor and/or an associated chipset and/or a special purpose microprocessor (e.g., an Application Specific Integrated Circuit (ASIC)), or the like. The processor 501 may also include on-board memory for caching purposes. The processor 501 may comprise a single processing unit or a plurality of processing units for performing different actions of the method flows according to embodiments of the disclosure.
In the RAM 503, various programs and data required for the operation of the electronic apparatus 500 are stored. The processor 501, ROM502, and RAM 503 are connected to each other by a bus 504. The processor 501 performs various operations of the method flow according to the embodiments of the present disclosure by executing programs in the ROM502 and/or the RAM 503. Note that the program may be stored in one or more memories other than the ROM502 and the RAM 503. The processor 501 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in the one or more memories.
According to an embodiment of the present disclosure, the electronic device 500 may also include an input/output (I/O) interface 505, the input/output (I/O) interface 505 also being connected to the bus 504. The electronic device 500 may also include one or more of an input section 506 including a keyboard, mouse, etc., an output section 507 including a Cathode Ray Tube (CRT), liquid Crystal Display (LCD), etc., and speaker, etc., a storage section 508 including a hard disk, etc., and a communication section 509 including a network interface card such as a LAN card, modem, etc., connected to the I/O interface 505. The communication section 509 performs communication processing via a network such as the internet. The drive 510 is also connected to the I/O interface 505 as needed. A removable medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 510 as needed so that a computer program read therefrom is mounted into the storage section 508 as needed.
According to embodiments of the present disclosure, the method flow according to embodiments of the present disclosure may be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable storage medium, the computer program comprising program code for performing the method shown in the flowcharts. In such an embodiment, the computer program may be downloaded and installed from a network via the communication portion 509, and/or installed from the removable media 511. The above-described functions defined in the system of the embodiments of the present disclosure are performed when the computer program is executed by the processor 501. The systems, devices, apparatus, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the disclosure.
The present disclosure also provides a computer-readable storage medium that may be included in the apparatus/device/system described in the above embodiments, or may exist alone without being assembled into the apparatus/device/system. The computer-readable storage medium carries one or more programs which, when executed, implement methods in accordance with embodiments of the present disclosure.
According to embodiments of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium. Such as, but not limited to, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this disclosure, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
For example, according to embodiments of the present disclosure, the computer-readable storage medium may include ROM 502 and/or RAM 503 and/or one or more memories other than ROM 502 and RAM 503 described above.
Embodiments of the present disclosure also include a computer program product comprising a computer program comprising program code for performing the methods provided by the embodiments of the present disclosure, the program code for causing an electronic device to implement the image processing methods provided by the embodiments of the present disclosure when the computer program product is run on the electronic device.
The above-described functions defined in the system/apparatus of the embodiments of the present disclosure are performed when the computer program is executed by the processor 501. The systems, apparatus, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the disclosure.
In one embodiment, the computer program may be based on a tangible storage medium such as an optical storage device, a magnetic storage device, or the like. In another embodiment, the computer program may also be transmitted, distributed, and downloaded and installed in the form of a signal on a network medium, and/or installed from a removable medium 511 via the communication portion 509. The computer program may comprise program code that is transmitted using any appropriate network medium, including but not limited to wireless, wireline, etc., or any suitable combination of the preceding.
According to embodiments of the present disclosure, program code for performing computer programs provided by embodiments of the present disclosure may be written in any combination of one or more programming languages, and in particular, such computer programs may be implemented in high-level procedural and/or object-oriented programming languages, and/or assembly/machine languages. Programming languages include, but are not limited to, such as Java, c++, python, "C" or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., connected via the Internet using an Internet service provider).
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions. Those skilled in the art will appreciate that the features recited in the various embodiments of the disclosure and/or in the claims may be combined in various combinations and/or combinations, even if such combinations or combinations are not explicitly recited in the disclosure. In particular, the features recited in the various embodiments of the present disclosure and/or the claims may be variously combined and/or combined without departing from the spirit and teachings of the present disclosure. All such combinations and/or combinations fall within the scope of the present disclosure.
The embodiments of the present disclosure are described above. These examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the embodiments cannot be used advantageously in combination. The scope of the disclosure is defined by the appended claims and equivalents thereof. Various alternatives and modifications can be made by those skilled in the art without departing from the scope of the disclosure, and such alternatives and modifications are intended to fall within the scope of the disclosure.