Method, equipment and the storage medium painted to line manuscript base pictureTechnical field
This disclosure relates to field of image processing, method, the equipment painted in particular to a kind of pair of line manuscript base pictureAnd storage medium.
Background technique
Caricature and cartoon making process are usually that producer first completes line original text, then carries out colouring to line original text and complete color original text.?During colouring, traditional scheme be usually utilize the auxiliary such as Adobe Photoshop software by hand to each region of line original text intoRow colouring.Therefore there are very high human costs for one caricature of production or animation, while longer Production Time also constrains productionThe distribution speed of product.
With the continuous development of information technology, had already appeared at present it is some using artificial intelligence approach to line original text carry out fromThe technology of dynamic colouring.The significantly larger than artificial colouring in efficiency of this automatic colouring technology, and advantage of lower cost, it is only necessary to oneFixed computer resource.Paintschainer is a kind of prior art, it can using the color manuscript base picture of specific style asWith reference to paint to new line original text.But due to the limitation of its method itself, its effect is very from artificial colouring result differenceIt greatly, can not be satisfactory；Its method efficiency is lower simultaneously, so that the program is difficult to carry out commercialization, especially carries out animation systemMake (per second to need comprising more than ten width color original texts).
Prior art Paintschainer has the following disadvantages:
1) coloring effect is poor.In the color original text that Paintschainer is completed, the boundary between different zones is relatively fuzzyyer, veryMore different zones should be described using different colours, but the color of its different zones is substantially coincident in these color original texts's.Meanwhile there is also some " ripples " for the color original text of certain generations, this is the artificial neural network that Paintschainer scheme usesThe problem of network model causes is that as caused by artificial nerve network model parameter is excessive and training sample is very few " over-fitting " is askedTopic.
2) colouring low efficiency.Since the artificial nerve network model parameter amount that Paintschainer scheme uses is very big,Calculating speed is very slow in practical application.
For above-mentioned problem, currently no effective solution has been proposed.
Summary of the invention
The embodiment of the present disclosure provides method, equipment and the storage medium that a kind of pair of line manuscript base picture is painted, so thatThe technical issues of solving the difference of coloring effect present in the existing technology painted to line original text and colouring low efficiency less.
According to the one aspect of the embodiment of the present disclosure, the method that a kind of pair of line manuscript base picture is painted is provided, comprising: obtainFirst Line manuscript base picture and the first reference picture are taken, wherein first reference picture includes to carry out to the First Line manuscript base pictureReferenced image information when colouring；And using the model based on machine learning training, it is based on first reference picture, it is rightThe First Line manuscript base picture carries out colouring processing.
According to the another aspect of the embodiment of the present disclosure, the equipment that a kind of pair of line manuscript base picture is painted is additionally provided,It include: processor；And memory, it is connected to the processor, for providing processing following processing step for the processorInstruction: First Line manuscript base picture and the first reference picture are obtained, wherein first reference picture includes to the First LineManuscript base picture image information referenced when being painted；And using the model based on machine learning training, it is based on described firstReference picture carries out colouring processing to the First Line manuscript base picture.
According to the another aspect of the embodiment of the present disclosure, the equipment that a kind of pair of line manuscript base picture is painted is additionally provided,It include: image collection module, for obtaining First Line manuscript base picture and the first reference picture, wherein the first reference picture packetContaining image information referenced when painting to the First Line manuscript base picture；And colouring module, for using based on machineThe model of learning training is based on first reference picture, carries out colouring processing to the First Line manuscript base picture.
In the embodiments of the present disclosure, using reference picture is based on, using the model based on machine learning training to line manuscript baseAs the mode painted, the technical effect for improving coloring effect and upper colour efficiency is realized, and then solve existing to lineThe technical issues of difference of coloring effect present in the technology that original text is painted and colouring low efficiency.
Detailed description of the invention
Attached drawing described herein is used to provide further understanding of the disclosure, constitutes part of this application, this public affairsThe illustrative embodiments and their description opened do not constitute the improper restriction to the disclosure for explaining the disclosure.In the accompanying drawings:
Fig. 1 shows the method painted to line manuscript base picture for realizing the first aspect according to the embodiment of the present disclosure 1Terminal hardware structure diagram；
Fig. 2 is the flow diagram for the method painted according to the embodiment of the present disclosure 1 to line manuscript base picture；
Fig. 3 is the further process in the method painted according to the embodiment of the present disclosure 1 to line manuscript base pictureSchematic diagram；
Fig. 4 shows the schematic diagram for the model painted according to the embodiment of the present disclosure 1 to line manuscript base picture；
Fig. 5 shows the line manuscript base of the model as the list of the convolutional layer structure of characteristic extracting module；
Fig. 6 shows the list of the convolutional layer structure of the reference picture characteristic extracting module of the model；
Fig. 7 shows the color manuscript base of the model as the list of the convolutional layer structure of feature coding module；
Fig. 8 shows the color manuscript base of the model as the list of the convolutional layer structure of generation module；
Fig. 9 shows the flow diagram for the operation being trained to model described in embodiment 1；
Figure 10 shows the flow diagram of the operation optimized to model described in embodiment 1；
Figure 11 shows the flow diagram for the operation assessed model described in embodiment 1；
Figure 12 shows the schematic diagram according to the equipment as described in example 2 painted to line manuscript base picture；And
Figure 13 shows the schematic diagram according to the equipment described in embodiment 3 painted to line manuscript base picture.
Specific embodiment
In order to make those skilled in the art more fully understand disclosure scheme, below in conjunction in the embodiment of the present disclosureThe technical solution in the embodiment of the present disclosure is clearly and completely described in attached drawing, it is clear that described embodiment is onlyThe embodiment of disclosure a part, instead of all the embodiments.Based on the embodiment in the disclosure, ordinary skill peopleMember's every other embodiment obtained without making creative work, all should belong to the model of disclosure protectionIt encloses.
It should be noted that the specification and claims of the disclosure and term " first " in above-mentioned attached drawing, "Two " etc. be to be used to distinguish similar objects, without being used to describe a particular order or precedence order.It should be understood that using in this wayData be interchangeable under appropriate circumstances, so as to embodiment of the disclosure described herein can in addition to illustrating herein orSequence other than those of description is implemented.In addition, term " includes " and " having " and their any deformation, it is intended that coverCover it is non-exclusive include, for example, the process, method, system, product or equipment for containing a series of steps or units are not necessarily limited toStep or unit those of is clearly listed, but may include be not clearly listed or for these process, methods, productOr other step or units that equipment is intrinsic.
Embodiment 1
According to the embodiment of the present disclosure, the embodiment of the method that a kind of pair of line manuscript base picture is painted is additionally provided, needs to illustrate, step shown in the flowchart of the accompanying drawings can hold in a computer system such as a set of computer executable instructionsRow, although also, logical order is shown in flow charts, and it in some cases, can be to be different from sequence hereinExecute shown or described step.
Embodiment of the method provided by the embodiment of the present application one can be in mobile terminal, terminal or similar fortuneIt calculates and is executed in device.It is (or mobile that Fig. 1 shows a kind of terminal for realizing the method painted to line manuscript base pictureEquipment) hardware block diagram.As shown in Figure 1, terminal 10 (or mobile device 10) may include one or more (figuresMiddle to use 102a, 102b ... ..., 102n is shown) (processor 102 can include but is not limited to microprocessor to processor 102The processing unit of MCU or programmable logic device FPGA etc.), memory 104 for storing data and be used for communication functionTransmitting device 106.It in addition to this, can also include: display, input/output interface (I/O interface), universal serial bus(USB) port (a port that can be used as in the port of I/O interface is included), network interface, power supply and/or camera.ThisField those of ordinary skill is appreciated that structure shown in FIG. 1 is only to illustrate, and does not cause to the structure of above-mentioned electronic deviceIt limits.For example, terminal 10 may also include than shown in Fig. 1 more perhaps less component or have with shown in Fig. 1Different configurations.
It is to be noted that said one or multiple processors 102 and/or other data processing circuits lead to hereinCan often " data processing circuit " be referred to as.The data processing circuit all or part of can be presented as software, hardware, firmwareOr any other combination.In addition, data processing circuit for single independent processing module or all or part of can be integrated to meterIn any one in other elements in calculation machine terminal 10 (or mobile device).As involved in the embodiment of the present application,The data processing circuit controls (such as the selection for the variable resistance end path connecting with interface) as a kind of processor.
Memory 104 can be used for storing the software program and module of application software, as in the embodiment of the present disclosure to lineCorresponding program instruction/the data storage device of the method that manuscript base picture is painted, processor 102 are stored in memory by operationSoftware program and module in 104 realize above-mentioned application program thereby executing various function application and data processingThe method painted to line manuscript base picture.Memory 104 may include high speed random access memory, may also include non-volatile memoriesDevice, such as one or more magnetic storage device, flash memory or other non-volatile solid state memories.In some instances, it depositsReservoir 104 can further comprise the memory remotely located relative to processor 102, these remote memories can pass through networkIt is connected to terminal 10.The example of above-mentioned network includes but is not limited to internet, intranet, local area network, moves and lead toLetter net and combinations thereof.
Transmitting device 106 is used to that data to be received or sent via a network.Above-mentioned network specific example may includeThe wireless network that the communication providers of terminal 10 provide.In an example, transmitting device 106 includes that a network is suitableOrchestration (Network Interface Controller, NIC), can be connected by base station with other network equipments so as toInternet is communicated.In an example, transmitting device 106 can be radio frequency (Radio Frequency, RF) module,For wirelessly being communicated with internet.
Display can such as touch-screen type liquid crystal display (LCD), the liquid crystal display aloow user withThe user interface of terminal 10 (or mobile device) interacts.
Under above-mentioned running environment, painting methods are carried out to line manuscript base picture this application provides as shown in Figure 2.Fig. 2 isThe flow chart for the method painted according to the first aspect of the embodiment of the present disclosure 1 to line manuscript base picture.With reference to Fig. 2 instituteShow, which comprises
S201: First Line manuscript base picture and the first reference picture are obtained；And
S202: using the model based on machine learning training, it is based on the first reference picture, First Line manuscript base picture is carried outColor processing.
Wherein, the first reference picture includes image information referenced when painting to First Line manuscript base picture.
As described in the background art, prior art Paintschainer is primarily present artificial neural network" over-fitting " problem caused by network model parameter is excessive and training sample is very few, and then lead to coloring effect difference and upper colour efficiencyLow problem.
In order to solve the technical problem, according to the offer of the first aspect of the present embodiment to line manuscript base as paintingMethod, by obtaining First Line manuscript base picture and the first reference picture, and using the model based on machine learning training, based on theOne reference picture carries out colouring processing to First Line manuscript base picture.
Due to including information referenced when painting to the First Line manuscript base picture, such as area in the first reference pictureThe information such as domain color and region division, therefore the disclosure can use the information of the first reference picture offer, to First LineManuscript base picture carries out colouring processing.To in this way, the present disclosure ensure that during painting to new line original text,By acquisition First Line manuscript base picture and the first reference picture, and using the model based on machine learning training, it is based on firstReference picture carries out colouring processing to First Line manuscript base picture.To improve line original text region resolution capability, enhance generation colorThe effect of original text.Meanwhile using machine learning training pattern, acquires a large amount of training samples and model is repeatedly trained, finallyAvoid " over-fitting " problem.
Wherein, the first reference picture includes image information referenced when painting to First Line manuscript base picture.Such as butIt is not limited to, which includes the information such as regional color and the region division of reference picture.
Optionally, as shown in figure 3, using the model based on machine learning training, it is based on the first reference picture, to First LineManuscript base picture carries out the operation of colouring processing, comprising:
Step S2011: using the first convolution model for including multiple convolutional layers, line manuscript base is extracted from First Line manuscript base pictureAs feature；
Step S2012, it using the second convolution model including multiple convolutional layers, extracts from the first reference picture with reference to figureAs feature；
Step S2013, using the third convolution model including multiple convolutional layers, First Line original text characteristics of image and ginseng are based onCharacteristics of image is examined, the color original text characteristics of image of coding is generated；And
Step S2014, using the Volume Four product module type including multiple convolutional layers, it is based on color original text characteristics of image, generates firstColor manuscript base picture is as the line manuscript base picture after colouring.
Wherein Fig. 4 shows the model based on machine learning training according to the first aspect of the present embodiment 1Schematic diagram.Refering to what is shown in Fig. 4, the model includes line manuscript base as characteristic extracting module 401 (corresponding to the first convolution model), referenceImage characteristics extraction module 402 (corresponding to the second convolution model), color manuscript base are as feature coding module 403 is (corresponding to third volumeProduct module type) and color manuscript base as generation module 404 (correspond to Volume Four product module type).
Wherein, line original text characteristic extracting module 401 is for receiving line manuscript base picture, and line manuscript base picture is extracted from line manuscript base pictureFeature.Wherein the line original text characteristics of image covers the letter such as structure and region division of line manuscript base picture in the form of one compared with low-dimensionalBreath.
Reference picture characteristic extracting module 402 extracts reference picture for receiving reference picture from reference pictureFeature.Wherein the reference picture feature contains the regional color of reference picture and colour gamut draws grading information.
Color manuscript base is used to receive line original text characteristics of image and reference picture feature as feature coding module 403, and is based onAs the color original text characteristics of image after coding, this feature includes for we for line original text characteristics of image and reference picture feature output three-dimensional matriceWish the region division and regional color information that color manuscript base picture has.
Color manuscript base is painted as generation module 404 is used to receive color original text characteristics of image using the convolution operation of convolutional layerLine manuscript base picture afterwards.
The disclosure, which passes through, utilizes multiple convolution models, extracts from First Line manuscript base picture and the first reference picture respectively correspondingCharacteristics of image, the color original text characteristics of image of coding is then generated according to the characteristics of image, utilizes the technology of " decoding re-encoding " formerReason generates the first color manuscript base picture.To improve line original text region resolution capability, to enhance the effect for generating color original text.
Optionally, the first convolution model (that is, line manuscript base is as characteristic extracting module 401) includes 10 convolutional layers and 5Down-sampled layer is wherein provided with a down-sampled layer between every two convolutional layer.The specific network layer of this model is arranged such as Fig. 5 instituteShow.Such as: the module design has a convolutional neural networks, this network is made of 10 convolutional layers and 5 down-sampled layers, whereinEach convolutional layer uses ReLU function as activation primitive.After we are toward this module input line manuscript base picture, dimension can be exportedDegree for 8 × 8 × 512 matrix as line original text characteristics of image, this feature covered in the form of one compared with low-dimensional line manuscript base asThe information such as structure and region division.
Optionally, the second convolution model (that is, reference picture characteristic extracting module 402) includes VGG19 convolutional neural networksStructure employed in model for the convolutional layer part of feature extraction.The specific network layer setting of this model is as shown in Figure 6.Such as: the reference picture for the expression colouring style that input user provides is adopted using 16 convolutional layers of VGG19 model and 5 dropsSample layer exports the matrix that dimension is 8 × 8 × 512 as reference picture feature, contains regional color and the region of reference pictureDraw grading information.
Optionally, third convolution model (that is, color manuscript base is as feature coding module 403) rises including 4 convolutional layers and 3 and adoptsSample layer, wherein there are two the convolutional layers for setting between every two liter sample level.The specific network layer of this model is arranged such as Fig. 7 instituteShow.Such as: then input line original text characteristics of image and reference picture feature adjust dimension and allow to utilize splicing(concatenate) mode merges it.Specific practice is using channel dimension as axis (axis), is 8 × 8 by dimension× 512 line original text characteristics of image and dimension be 8 × 8 × 512 reference picture merging features at 8 × 8 × 1024 three-dimensional matrice.It rises sample level using 4 convolutional layers and 3 after fusion to be encoded, wherein each convolutional layer uses ReLU function as activationFunction, for output three-dimensional matrice as the color original text characteristics of image after coding, this feature includes it is desirable that the area that color manuscript base picture hasDomain divides and regional color information.
Optionally, Volume Four product module type (that is, color manuscript base is as generation module 404) includes that 8 convolutional layers and 2 rise samplingLayer, wherein there are two the convolutional layers for setting between every two liter sample level.The specific network layer of this model is arranged such as Fig. 8 instituteShow.Such as: the color original text characteristics of image after input coding rises sample level using 8 convolutional layers and 2 and carries out the generation of color manuscript base picture,Wherein each convolutional layer uses ReLU function as activation primitive.Finally network is exported further according to the size of line manuscript base pictureImage down or amplification (resize), the image after colouring can be obtained.
It is alternatively possible to be trained by following operation to the model based on machine learning training.Fig. 9 is to the machine of being based onThe flow chart that the model of device learning training is trained.
S901, the second color manuscript base picture and the second reference picture are obtained；
S902, the second line manuscript base picture is generated based on the second acquired color manuscript base picture；
S903, it is based on the second line manuscript base picture and the second reference picture, using the model based on machine learning training, to theTwo wires manuscript base picture carries out colouring processing；
S904, to the second color manuscript base picture and colouring, treated that the second line manuscript base picture is compared；And
S905, using it is after comparison as a result, to based on machine learning training model optimize.
The disclosure is trained the model based on machine learning training by above-mentioned 5 steps.For example, can set in advanceA line manuscript base is set as generation module, can extract using edge detection algorithm according to the color manuscript base picture of input and export edgeImage is as corresponding line manuscript base picture.Another example is: the method according to the first aspect of the present embodiment, first initialization instituteThere is the parameter of network layer, for reference picture characteristic extracting module, we used the model parameters of pre-training on ImageNetAs initial parameter, other network layer parameters are initialized using normal distribution random number.Then will comprising color manuscript base picture andThe training sample of reference picture (that is, the second color manuscript base picture recited above and second reference picture) is divided into multiple groups, in batchesIn model used in secondary input present patent application technology, the color manuscript base picture that is generated is (that is, colouring treated the second line original textImage), and treated that the second line manuscript base picture is compared to the second color manuscript base picture and colouring.Then next batch is inputted againTraining sample can be obtained by the model after training after circulation is multiple.Using after comparison as a result, to based on machine learning trainingModel optimize.
Optionally, to the second color manuscript base picture and colouring treated operation that the second line manuscript base picture is compared, includingCalculate the L1 space length between the second color manuscript base picture and colouring treated the second line manuscript base picture, and to being based on engineeringThe operation that the model of training optimizes is practised, flow chart is as shown in Figure 10.
S1001: using L1 space length as loss function, the gradient of loss function is calculated；And
S1002: being based on gradient, is optimized according to stochastic gradient descent principle to the model based on machine learning training.
After the present embodiment generates color manuscript base picture by above-mentioned module, to the second color manuscript base picture and colouring, treated theThe operation that two wires manuscript base picture is compared, including calculate between the second color manuscript base picture and colouring treated the second line manuscript base pictureL1 space length, and to the operation that optimizes of model based on machine learning training.Such as: the color manuscript base generatedAs after, the L1 space length between the color manuscript base picture of generation and the original color manuscript base picture of input is calculated as loss function, is calculatedThe gradient of the loss function simultaneously optimizes according to stochastic gradient descent principle (Stochastic Gradient Descent, i.e. SGD)Network parameter.
Optionally, optimization includes at least one of carrying out the following processing to the model of machine learning training: parameter two-valueChange, reasoning and beta pruning.Such as but be not limited only to, which can make the parameter of model reduction 50% and improve 10 speedsDegree.
Optionally, the model based on machine learning training is assessed by following operation, Figure 11 is to based on machineThe flow chart that the model of learning training is assessed.
S1101: third line manuscript base picture, third reference picture are obtained and with third line manuscript base as corresponding true color manuscript basePicture；
S1102: being based on third line manuscript base picture and third reference picture, is generated using the model based on machine learning trainingThird color manuscript base picture；
S1103: the first similarity and third color manuscript base picture and the of third color manuscript base picture and true color manuscript base picture are calculatedSecond similarity of three reference pictures；
S1104: the sum of the similarity of the first similarity and the second similarity is calculated；And
S1105: using the sum of similarity calculated, whether assessment third color original text is pressed under the premise of guaranteeing that content is constantIt paints according to third reference picture.
Such as: under evaluation profile, input line manuscript base picture and reference picture export color manuscript base picture by network processes, completeAt colouring task.We take a variety of image quality measure algorithms and evaluate the color manuscript base picture of output.Utilize these figuresImage quality amount assessment algorithm can calculate the similarity for generating color manuscript base picture and true color manuscript base picture and similar with reference pictureDegree, with the sum of the two similarities assess generate color manuscript base seem it is no guarantee content it is constant under the premise of by reference pictureColor, so that it is guaranteed that network can export up-to-standard color manuscript base picture.
Optionally, the first similarity is any one similarity in the following terms: SSIM (StrcturalSimilarity, structural similarity), FSIM (Feature similarity, characteristic similarity) and MS-SSIM (Multi-Scale extension ofSSIM, multi-level structure similarity).
In addition, the second similarity is any one similarity in the following terms: SSIM (StrcturalSimilarity, structural similarity), FSIM (Feature similarity, characteristic similarity) and MS-SSIM (Multi-Scale extension ofSSIM, multi-level structure similarity).
Method described in first aspect to the present embodiment utilizes " decoding by a kind of artificial neural network structureThe technical principle of re-encoding ", improves line original text region resolution capability, to enhance the effect for generating color original text.Joined using modelThe optimisation technique of the artificial intelligence fields such as number binaryzation, model reasoning and model beta pruning, greatly reduces artificial neural networkParameter amount improves the efficiency of colouring；We acquire a large amount of training samples and are repeatedly trained to model simultaneously, finally avoid" over-fitting " problem.And then solve it is existing it is poor to line original text coloring effect, paint low efficiency the technical issues of.
In addition, according to the second aspect of the present embodiment, providing a kind of storage medium 102, storage medium with reference to Fig. 1Program including storage, wherein the method described in equipment any of the above one where control storage medium in program operation.
It should be noted that for the various method embodiments described above, for simple description, therefore, it is stated as a series ofCombination of actions, but those skilled in the art should understand that, the disclosure is not limited by the described action sequence becauseAccording to the disclosure, some steps may be performed in other sequences or simultaneously.Secondly, those skilled in the art should also knowIt knows, the embodiments described in the specification are all preferred embodiments, the related actions and modules not necessarily disclosureIt is necessary.
Through the above description of the embodiments, those skilled in the art can be understood that according to above-mentioned implementationThe method of example can be realized by means of software and necessary general hardware platform, naturally it is also possible to by hardware, but it is very muchIn the case of the former be more preferably embodiment.Based on this understanding, the technical solution of the disclosure is substantially in other words to existingThe part that technology contributes can be embodied in the form of software products, which is stored in a storageIn medium (such as ROM/RAM, magnetic disk, CD), including some instructions are used so that a terminal device (can be mobile phone, calculateMachine, server or network equipment etc.) execute method described in each embodiment of the disclosure.
Embodiment 2
According to the embodiment of the present disclosure, a kind of line original text automatic colouring equipment 1200 is additionally provided.The equipment is and embodiment 1The corresponding equipment of method described in one aspect.With reference to shown in Figure 12, which includes: processor 1210；And memory1220, it is connect with processor, for providing the instruction for handling following processing step for processor:
Obtain First Line manuscript base picture and the first reference picture, wherein the first reference picture include to First Line manuscript base picture intoReferenced image information when row colouring；And
Using the model based on machine learning training, it is based on the first reference picture, First Line manuscript base picture is carried out at colouringReason.
Optionally, using the model based on machine learning training, it is based on first reference picture, to the First Line original textImage carries out the operation of colouring processing, comprising: using the first convolution model for including multiple convolutional layers, from the First Line manuscript baseLine original text characteristics of image is extracted as in；Using the second convolution model including multiple convolutional layers, mentioned from first reference pictureTake reference picture feature；Using the third convolution model including multiple convolutional layers, based on the First Line original text characteristics of image andThe reference picture feature generates the color original text characteristics of image of coding, wherein the color original text characteristics of image includes that color manuscript base picture hasRegion division and regional color information；And using the Volume Four product module type for including multiple convolutional layers, it is based on the color original textCharacteristics of image generates the first color manuscript base picture as the line manuscript base picture after colouring.
Optionally, first convolution model includes 10 convolutional layers and 5 down-sampled layers, is wherein rolled up described in every twoThe down-sampled layer is provided between lamination.
Optionally, second convolution model includes mentioning employed in VGG19 convolutional neural networks model for featureThe structure of the convolutional layer part taken.
Optionally, the third convolution model includes that 4 convolutional layers and 3 rise sample level, and wherein liter described in every two is adoptedThere are two the convolutional layers for setting between sample layer.
Optionally, the Volume Four product module type includes that 8 convolutional layers and 2 rise sample level, and wherein liter described in every two is adoptedThere are two the convolutional layers for setting between sample layer.
Optionally, further include being trained by following operation to the model based on machine learning training: obtaining theTwo color manuscript base pictures and the second reference picture；The second line manuscript base picture is generated based on the second acquired color manuscript base picture；Based on describedSecond line manuscript base picture and second reference picture, using the model based on machine learning training, to second lineManuscript base picture carries out colouring processing；To the second color manuscript base picture and colouring, treated that the second line manuscript base picture comparesIt is right；And using after the comparison as a result, being optimized to the model based on machine learning training.
Optionally, to the second color manuscript base picture and colouring treated behaviour that the second line manuscript base picture is comparedMake, the L1 space length between treated including calculating the second color manuscript base picture and colouring the second line manuscript base picture,And to the operation that the model based on machine learning training optimizes, comprising: using the L1 space length as lossFunction calculates the gradient of the loss function；And it is based on the gradient, machine is based on to described according to stochastic gradient descent principleThe model of device learning training optimizes.
Optionally, to it is described optimization include to the machine learning training model carry out the following processing at least oneKind: parameter binaryzation, reasoning and beta pruning.
Optionally, the model based on machine learning training is assessed by following operation: obtains third line original textImage, third reference picture and with the third line manuscript base as corresponding true color manuscript base picture；Based on the third line manuscript basePicture and the third reference picture utilize the model based on machine learning training to generate third color manuscript base picture；Calculate instituteThe first similarity and the third color manuscript base picture for stating third color manuscript base picture and the true color manuscript base picture are joined with the thirdExamine the second similarity of image；Calculate the sum of the similarity of first similarity Yu second similarity；And utilize instituteThe sum of similarity of calculating, assess the third color original text whether guarantee content it is constant under the premise of according to the third with reference to figureAs colouring.
Optionally, first similarity is any one similarity in the following terms: SSIM, FSIM andMS-SSIM。
Optionally, second similarity is any one similarity in the following terms: SSIM, FSIM andMS-SSIM。
To the present embodiment equipment during painting to new line original text, by obtain First Line manuscript base picture withAnd first reference picture, and using the model based on machine learning training, it is based on the first reference picture, to First Line manuscript base pictureCarry out colouring processing.The technical principle for utilizing " decoding re-encoding ", improves line original text region resolution capability, to enhance generationThe effect of color original text.Meanwhile using machine learning training pattern, acquires a large amount of training samples and model is repeatedly trained, most" over-fitting " problem is avoided eventually.
Embodiment 3
According to the embodiment of the present disclosure, a kind of line original text automatic colouring equipment 1300 is additionally provided.The equipment is and embodiment 1The corresponding equipment of method described in one aspect.With reference to shown in Figure 13, which includes: image collection module 1310, for obtainingFirst Line manuscript base picture and the first reference picture are taken, wherein first reference picture includes to carry out to the First Line manuscript base pictureReferenced image information when colouring；And colouring module 1320, for being based on institute using the model based on machine learning trainingThe first reference picture is stated, colouring processing is carried out to the First Line manuscript base picture.
Optionally, using the model based on machine learning training, colouring module 1320 includes: the first submodule, for benefitWith the first convolution model including multiple convolutional layers, line original text characteristics of image is extracted from the First Line manuscript base picture；Second submoduleBlock, for extracting reference picture feature from first reference picture using the second convolution model for including multiple convolutional layers；Third submodule, for using the third convolution model for including multiple convolutional layers, based on the First Line original text characteristics of image andThe reference picture feature generates the color original text characteristics of image of coding, wherein the color original text characteristics of image includes that color manuscript base picture hasRegion division and regional color information；And the 4th submodule, for utilizing the Volume Four product module for including multiple convolutional layersType is based on the color original text characteristics of image, generates the first color manuscript base picture as the line manuscript base picture after colouring.
Optionally, further include training module, for by following submodule to it is described based on machine learning training modelIt is trained: the 5th submodule, for obtaining the second color manuscript base picture and the second reference picture；6th submodule, for being based onThe second acquired color manuscript base picture generates the second line manuscript base picture；7th submodule, for based on the second line manuscript base picture andSecond reference picture carries out at colouring the second line manuscript base picture using the model based on machine learning trainingReason；8th submodule, for treated that the second line manuscript base picture is compared to the second color manuscript base picture and colouring；And the 9th submodule, for using after the comparison as a result, being optimized to the model based on machine learning training.
Optionally, the 8th submodule includes first unit, after calculating the second color manuscript base picture and colouring processingThe second line manuscript base picture between L1 space length and the 9th submodule, comprising: second unit, for by the L1Space length calculates the gradient of the loss function as loss function；And third unit, for being based on the gradient, rootThe model based on machine learning training is optimized according to stochastic gradient descent principle.
Optionally, third unit includes the first subelement, for carrying out following place to the model of machine learning trainingAt least one of reason: parameter binaryzation, reasoning and beta pruning.
Optionally, further include evaluation module, the model based on machine learning training is carried out by following submoduleAssessment: the tenth submodule, for obtaining third line manuscript base picture, third reference picture and with the third line manuscript base as correspondingTrue color manuscript base picture；11st submodule utilizes institute for being based on the third line manuscript base picture and the third reference pictureIt states the model based on machine learning training and generates third color manuscript base picture；12nd submodule, for calculating the third color manuscript baseAs second with the first similarity of the true color manuscript base picture and the third color manuscript base picture and the third reference pictureSimilarity；13rd submodule, for calculating the sum of the similarity of first similarity Yu second similarity；And the14 submodules, for assessing the third color original text whether before guaranteeing that content is constant using the sum of similarity calculatedIt puts and paints according to the third reference picture.
To the present embodiment equipment during painting to new line original text, by obtain First Line manuscript base picture withAnd first reference picture, and using the model based on machine learning training, it is based on the first reference picture, to First Line manuscript base pictureCarry out colouring processing.The technical principle for utilizing " decoding re-encoding ", improves line original text region resolution capability, to enhance generationThe effect of color original text.Meanwhile using machine learning training pattern, acquires a large amount of training samples and model is repeatedly trained, most" over-fitting " problem is avoided eventually.
Above-mentioned embodiment of the present disclosure serial number is for illustration only, does not represent the advantages or disadvantages of the embodiments.
In above-described embodiment of the disclosure, all emphasizes particularly on different fields to the description of each embodiment, do not have in some embodimentThe part of detailed description, reference can be made to the related descriptions of other embodiments.
In several embodiments provided herein, it should be understood that disclosed technology contents can pass through othersMode is realized.Wherein, the apparatus embodiments described above are merely exemplary, such as the division of the unit, onlyA kind of logical function partition, there may be another division manner in actual implementation, for example, multiple units or components can combine orPerson is desirably integrated into another system, or some features can be ignored or not executed.Another point, shown or discussed is mutualBetween coupling, direct-coupling or communication connection can be through some interfaces, the INDIRECT COUPLING or communication link of unit or moduleIt connects, can be electrical or other forms.
The unit as illustrated by the separation member may or may not be physically separated, aobvious as unitThe component shown may or may not be physical unit, it can and it is in one place, or may be distributed over multipleIn network unit.It can select some or all of unit therein according to the actual needs to realize the mesh of this embodiment scheme's.
It, can also be in addition, each functional unit in each embodiment of the disclosure can integrate in one processing unitIt is that each unit physically exists alone, can also be integrated in one unit with two or more units.Above-mentioned integrated listMember both can take the form of hardware realization, can also realize in the form of software functional units.
If the integrated unit is realized in the form of SFU software functional unit and sells or use as independent productWhen, it can store in a computer readable storage medium.Based on this understanding, the technical solution of the disclosure is substantiallyThe all or part of the part that contributes to existing technology or the technical solution can be in the form of software products in other wordsIt embodies, which is stored in a storage medium, including some instructions are used so that a computerEquipment (can for personal computer, server or network equipment etc.) execute each embodiment the method for the disclosure whole orPart steps.And storage medium above-mentioned includes: that USB flash disk, read-only memory (ROM, Read-Only Memory), arbitrary access are depositedVarious Jie that can store program code such as reservoir (RAM, RandomAccess Memory), mobile hard disk, magnetic or diskMatter.
The above is only the preferred embodiment of the disclosure, it is noted that for the ordinary skill people of the artFor member, under the premise of not departing from disclosure principle, several improvements and modifications can also be made, these improvements and modifications are also answeredIt is considered as the protection scope of the disclosure.