Invention content
The embodiment of the present application proposes the method and apparatus for testing user interface.
In a first aspect, the embodiment of the present application provides a kind of method for testing user interface, this method includes：It obtainsThe sectional drawing of user interface to be tested；Determine whether the sectional drawing of user interface to be tested meets preset condition；In response to determining to meetThe sectional drawing of user interface to be tested is input to global test model by preset condition, is obtained corresponding with user interface to be testedGlobal test result；And at least one regional area is partitioned into from the sectional drawing of user interface to be tested, by what is be partitioned intoAt least one regional area is input to local test model trained in advance, obtains part corresponding with user interface to be tested and surveysTest result, wherein, each regional area includes a character string.
In some embodiments, determine whether the sectional drawing of user interface to be tested meets preset condition, including：It performs followingAt least one operation：At least one of the sectional drawing of user interface to be tested word is identified using optical character identification OCR techniqueSymbol goes here and there and matches at least one character string identified in default error character set of strings；Utilize color mouldType determines the color category included by the sectional drawing of user interface to be tested；Based at least one of above operation as a result, determining to treatWhether the sectional drawing of test user interface meets preset condition, wherein, preset condition includes at least one of following：Character string is defaultUnsuccessful, color category number is matched in error character set of strings more than preset number.
In some embodiments, at least one regional area is partitioned into from the sectional drawing of user interface to be tested, including：It obtainsThe DOM Document Object Model DOM of user interface to be tested is taken, wherein, the DOM of user interface to be tested includes at least one firstThe location information of regional area, wherein, each First partial region includes a character string；According at least one First partialThe instruction of the location information in region is partitioned at least one First partial region from the sectional drawing of user interface to be tested, asAt least one regional area.
In some embodiments, the position of at least one second regional area is further included in the DOM of user interface to be testedInformation, wherein, each second regional area includes multiple character strings；And it is partitioned into from the sectional drawing of user interface to be testedAt least one regional area, further includes：According to the instruction of the location information of at least one second regional area, from user to be testedAt least one second regional area is partitioned into the sectional drawing at interface；For each second game at least one second regional areaMultiple character strings in second regional area are identified and from second regional area point in portion region using OCR techniqueThe region where each character string in the multiple character strings identified is cut out, as multiple regional areas.
In some embodiments, global test result includes the position letter in global type and region corresponding with global typeBreath, local test result include local type and the location information in region corresponding with local type.
In some embodiments, global type includes at least one of following：Global normal type, the overall situation are left blank type, completeOffice's keyboard Exception Type, local type include at least one of following：Character normal type, character overlap type, character background mistakeCross Exception Type, character edge type of barrier.
In some embodiments, global test model includes at least one of following：The overall situation is left blank test model, global keyboardAbnormality test model, local test model include at least one of following：Character overlap test model, the transition of character background are surveyed extremelyDie trial type, character edge block test model.
In some embodiments, global test model is trained as follows obtains：Acquisition belongs to global normalThe sectional drawing at the sample of users interface of type and with belong to global normal type the corresponding global test in sample of users interface as a result,As global positive sample；Obtain the sectional drawing at the sample of users interface for belonging to global type of leaving blank and with belonging to global type of leaving blankThe corresponding global test in sample of users interface is as a result, as the first global negative sample；Acquisition belongs to global keyboard Exception TypeThe sectional drawing at sample of users interface and with belonging to the corresponding global test in sample of users interface of global keyboard Exception Type as a result, makingFor the second global negative sample；Using deep learning method, based on global positive sample and the first global negative sample to preset firstConvolutional neural networks are trained, and are obtained the overall situation and are left blank test model；Using deep learning method, based on global positive sample andTwo global negative samples are trained preset second convolutional neural networks, obtain global keyboard abnormality test model.
In some embodiments, global negative sample generates as follows：First default textures are covered in categoryIn on the sectional drawing at the sample of users interface of global normal type, generation belongs to cutting for the sample of users interface of global type of leaving blankFigure；Default keyboard exception textures are covered on the sectional drawing at the sample of users interface for belonging to global normal type, generation belongs to completeThe sectional drawing at the sample of users interface of office's keyboard Exception Type.
In some embodiments, this method further includes：To the sectional drawing at sample of users interface, the category for belonging to global normal typeIt is carried out in the sectional drawing at the sample of users interface of global type of leaving blank and the sectional drawing at the sample of users interface for belonging to global normal typeThe processing of figure isomery generates multiple global positive samples, multiple first global negative samples and multiple second global negative samples.
In some embodiments, local test model is trained as follows obtains：It is normal that acquisition belongs to characterThe sample regional area of type and with belonging to the corresponding local test of sample regional area of character normal type as a result, as officePortion's positive sample；Obtain the sample regional area for belonging to character overlap type and the sample regional area with belonging to character overlap typeCorresponding local test is as a result, as First partial negative sample；Obtain the sample part for belonging to character background transition Exception TypeRegion and with belonging to the corresponding local test of sample regional area of character background transition Exception Type as a result, as the second partNegative sample；Obtain the sample regional area for belonging to character edge type of barrier and the sample office with belonging to character edge type of barrierThe corresponding local test in portion region is as a result, as third part negative sample；Using deep learning method, based on local positive sample andFirst partial negative sample is trained preset third convolutional neural networks, obtains character overlap test model；Utilize depthLearning method is trained preset Volume Four product neural network based on local positive sample and the second local negative sample, obtainsCharacter background transition abnormality test model；Using deep learning method, based on local positive sample and third part negative sample to pre-If the 5th convolutional neural networks be trained, obtain character edge and block test model.
In some embodiments, local negative sample generates as follows：Preset characters textures are covered in categoryIn on the sample regional area of character normal type, generation belongs to the sample regional area of character overlap type；By default figure layerIt is covered on the sample regional area for belonging to character normal type, generation belongs to the sample part of character background transition Exception TypeRegion；Second default textures are covered on the edge for the sample regional area for belonging to character normal type, generation belongs to characterBlock the sample regional area of test model in edge.
In some embodiments, this method further includes：To belonging to the sample regional area of character normal type, belonging to characterThe sample regional area of overlapping type, the sample regional area for belonging to character background transition Exception Type and belong to character edge screeningThe sample regional area for keeping off test model carries out figure isomery processing, generates multiple local positive samples, multiple First partials bear sampleOriginally, the multiple second local negative samples and multiple third parts negative sample.
Second aspect, the embodiment of the present application provide a kind of device for being used to test user interface, which includes：It obtainsUnit is configured to obtain the sectional drawing of user interface to be tested；Determination unit is configured to determine cutting for user interface to be testedWhether figure meets preset condition；Test cell is configured in response to determining to meet preset condition, by user interface to be testedSectional drawing is input to global test model, obtains global test result corresponding with user interface to be tested；And it is tried out to be measuredAt least one regional area is partitioned into the sectional drawing at family interface, at least one regional area being partitioned into is input to advance instructionExperienced local test model obtains local test corresponding with user interface to be tested as a result, wherein, being wrapped in each regional areaInclude a character string.
In some embodiments, determination unit includes：Execution module is configured to carry out at least one of following operation：ProfitAt least one of the sectional drawing of user interface to be tested character string is identified with optical character identification OCR technique and will be knownAt least one character string not gone out is matched in default error character set of strings；User to be tested is determined using color modelColor category included by the sectional drawing at interface；Determining module is configured to based at least one of above operation as a result, determining to treatWhether the sectional drawing of test user interface meets preset condition, wherein, preset condition includes at least one of following：Character string is defaultUnsuccessful, color category number is matched in error character set of strings more than preset number.
In some embodiments, test cell includes：DOM Document Object Model acquisition module is configured to obtain to be measured on probationThe DOM Document Object Model DOM at family interface, wherein, the DOM of user interface to be tested includes at least one First partial regionLocation information, wherein, each First partial region includes a character string；First partial region segmentation module, is configured toAccording to the instruction of the location information at least one First partial region, at least one is partitioned into from the sectional drawing of user interface to be testedA First partial region, as at least one regional area.
In some embodiments, the position of at least one second regional area is further included in the DOM of user interface to be testedInformation, wherein, each second regional area includes multiple character strings；And test cell further includes：Second regional area pointModule is cut, is configured to the instruction of the location information according at least one second regional area, from cutting for user interface to be testedAt least one second regional area is partitioned into figure；Regional area divides module, is configured to for at least one second partEach second regional area in region identifies multiple character strings in second regional area, Yi Jicong using OCR techniqueThe region where each character string in the multiple character strings identified is partitioned into second regional area, as multiple officesPortion region.
In some embodiments, global test result includes the position letter in global type and region corresponding with global typeBreath, local test result include local type and the location information in region corresponding with local type.
In some embodiments, global type includes at least one of following：Global normal type, the overall situation are left blank type, completeOffice's keyboard Exception Type, local type include at least one of following：Character normal type, character overlap type, character background mistakeCross Exception Type, character edge type of barrier.
In some embodiments, global test model includes at least one of following：The overall situation is left blank test model, global keyboardAbnormality test model, local test model include at least one of following：Character overlap test model, the transition of character background are surveyed extremelyDie trial type, character edge block test model.
In some embodiments, which further includes global test model training unit, global test module training unitIncluding：Global positive sample acquisition module, be configured to obtain belong to global normal type sample of users interface sectional drawing and withBelong to the corresponding global test in sample of users interface of global normal type as a result, as global positive sample；First global negative sampleThis acquisition module is configured to obtain the sectional drawing at the sample of users interface for belonging to global type of leaving blank and leaves blank class with belonging to the overall situationThe corresponding global test in sample of users interface of type is as a result, as the first global negative sample；Second global negative sample acquisition module,It is configured to obtain the sectional drawing at the sample of users interface for belonging to global keyboard Exception Type and with belonging to global keyboard Exception TypeThe corresponding global test in sample of users interface as a result, as the second global negative sample；The overall situation is left blank test model training module,It is configured to using deep learning method, based on global positive sample and the first global negative sample to preset first convolution nerve netNetwork is trained, and is obtained the overall situation and is left blank test model；Global keyboard abnormality test model training module is configured to utilize depthLearning method is trained preset second convolutional neural networks based on global positive sample and the second global negative sample, obtainsGlobal keyboard abnormality test model.
In some embodiments, which further includes global sample generation unit, and global sample generation unit includes：FirstGlobal negative sample generation module is configured to for the first default textures to be covered in the sample of users interface for belonging to global normal typeSectional drawing on, generation belongs to the sectional drawing at the sample of users interface of global type of leaving blank；Second global negative sample generation module, configurationFor default keyboard exception textures to be covered in the sectional drawing at the sample of users interface for belonging to global normal type, generation belongs to completeThe sectional drawing at the sample of users interface of office's keyboard Exception Type.
In some embodiments, global sample generation unit further includes：First figure isomery processing module, is configured to pairBelong to the sectional drawing at the sample of users interface of global normal type, the sample of users interface for belonging to global type of leaving blank sectional drawing and categorySectional drawing in the sample of users interface of global normal type carries out figure isomery processing, generates multiple global positive samples, Duo GeOne global negative sample and multiple second global negative samples.
In some embodiments, which further includes local test model training unit, local test module training unitIncluding：Local positive sample acquisition module is configured to acquisition and belongs to the sample regional area of character normal type and with belonging to wordThe corresponding local test of sample regional area of normal type is accorded with as a result, as local positive sample；First partial negative sample obtainsModule is configured to obtain the sample regional area for belonging to character overlap type and the sample part with belonging to character overlap typeThe corresponding local test in region is as a result, as First partial negative sample；Second local negative sample acquisition module, is configured to obtainThe sample regional area for belonging to character background transition Exception Type and the sample part with belonging to character background transition Exception TypeThe corresponding local test in region is as a result, as the second local negative sample；Third part negative sample acquisition module, is configured to obtainSample regional area and the sample regional area with belonging to character edge type of barrier for belonging to character edge type of barrier are correspondingLocal test as a result, as third part negative sample；Character overlap test model training module is configured to utilize depthLearning method is trained preset third convolutional neural networks based on local positive sample and First partial negative sample, obtains wordAccord with overlap test model；Character background transition abnormality test model training module is configured to, using deep learning method, be based onLocal positive sample and the second local negative sample are trained preset Volume Four product neural network, and it is different to obtain character background transitionNormal test model；Character edge blocks test model training module, is configured to using deep learning method, based on the positive sample in partThis and third part negative sample are trained preset 5th convolutional neural networks, obtain character edge and block test model.
In some embodiments, which further includes fractional sample generation unit, and fractional sample generation unit includes：FirstLocal negative sample generation module is configured to for preset characters textures to be covered in the sample regional area for belonging to character normal typeOn, generation belongs to the sample regional area of character overlap type；Second local negative sample generation module, is configured to presetFigure layer is covered on the sample regional area for belonging to character normal type, and generation belongs to the sample of character background transition Exception TypeRegional area；Third part negative sample generation module, is configured to that the second default textures will be covered in and belongs to the normal class of characterOn the edge of the sample regional area of type, generation belongs to the sample regional area that character edge blocks test model.
In some embodiments, fractional sample generation unit further includes：Second graph isomery processing module, is configured to pairThe sample regional area that belongs to character normal type, belongs to character background mistake at the sample regional area for belonging to character overlap typeIt crosses the sample regional area of Exception Type and belongs to the sample regional area progress figure isomery that character edge blocks test modelIt is locally negative to generate multiple local positive samples, multiple First partial negative samples, multiple second local negative samples and multiple thirds for processingSample.
The third aspect, the embodiment of the present application provide a kind of electronic equipment, which includes：One or more processingDevice；Storage device, for storing one or more programs；When one or more programs are executed by one or more processors, makeObtain method of the one or more processors realization as described in realization method any in first aspect.
Fourth aspect, the embodiment of the present application provide a kind of computer-readable medium, are stored thereon with computer program, shouldThe method as described in realization method any in first aspect is realized when computer program is executed by processor.
Method and apparatus provided by the embodiments of the present application for testing user interface, by obtaining user interface to be testedSectional drawing, in order to determine user interface to be tested sectional drawing whether meet preset condition；In the feelings for determining to meet preset conditionUnder condition, the sectional drawing of user interface to be tested is input to global test model first, so as to obtain and user interface pair to be testedThe global test result answered；Then at least one regional area is partitioned into from the sectional drawing of user interface to be tested, will be dividedAt least one regional area gone out is input to local test model trained in advance, corresponding with user interface to be tested so as to obtainLocal test result.Using deep learning method, user interface is carried out with reference to global test model and local test modelTest, does not need to tester's artificial judgment, not only saves human cost, also improve the testing efficiency to user interface.
Specific embodiment
The application is described in further detail with reference to the accompanying drawings and examples.It is understood that this place is retouchedThe specific embodiment stated is used only for explaining related invention rather than the restriction to the invention.It also should be noted that in order toConvenient for description, illustrated only in attached drawing and invent relevant part with related.
It should be noted that in the absence of conflict, the feature in embodiment and embodiment in the application can phaseMutually combination.The application is described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.
Fig. 1 shows the method for being used to test user interface that can apply the application or the dress for testing user interfaceThe exemplary system architecture 100 for the embodiment put.
As shown in Figure 1, system architecture 100 can include terminal device 101,102,103, network 104 and server 105.Network 104 between terminal device 101,102,103 and server 105 provide communication link medium.Network 104 can be withIncluding various connection types, such as wired, wireless communication link or fiber optic cables etc..
User can be interacted with using terminal equipment 101,102,103 by network 104 with server 105, to receive or send outSend message etc..Various client applications, such as the application of shopping class, searching class can be installed on terminal device 101,102,103Using, instant messaging tools, mailbox client, social platform software etc..
Terminal device 101,102,103 can be various electronic equipments, including but not limited to smart mobile phone, tablet computer,Pocket computer on knee and desktop computer etc..
Server 105 can be to provide the server of various services, such as to user interface that user interface is testedTest server.Ui testing server can carry out sectional drawing of user interface to be tested for receiving etc. analyzing etc.Reason, and generate handling result (such as global test result corresponding with user interface to be tested and local test result).
It should be noted that the method for being used to test user interface that the embodiment of the present application is provided is generally by server105 perform, and correspondingly, the device for testing user interface is generally positioned in server 105.
It should be understood that the number of the terminal device, network and server in Fig. 1 is only schematical.According to realization needWill, can have any number of terminal device, network and server.
With continued reference to Fig. 2, it illustrates one embodiment for being used to test the method for user interface according to the applicationFlow 200.This is used for the method for testing user interface, includes the following steps：
Step 201, the sectional drawing of user interface to be tested is obtained.
In the present embodiment, it is (such as shown in FIG. 1 for testing the electronic equipment of the method for user interface operation thereonServer 105) it can from terminal device, (such as terminal shown in FIG. 1 be set by wired connection mode or radio connectionStandby 101,102,103) obtain the sectional drawing of user interface to be tested.In general, terminal device can be the electronics for having the function of sectional drawingEquipment is equipped with various client applications, such as the application of shopping class, searching class application, instant messaging tools, mailbox client thereonEnd, social platform software etc..Terminal device can show the user interface of client application during client application is run,Terminal device can utilize sectional drawing function to intercept shown user interface at this time.Wherein, user interface refers to the operation of userInterface, also referred to as man-machine interface, user can be interacted by user interface and client application.
Step 202, determine whether the sectional drawing of user interface to be tested meets preset condition.
In the present embodiment, the sectional drawing based on the user interface to be tested acquired in step 201, electronic equipment can determineWhether the sectional drawing of user interface to be tested meets preset condition, in the case where determining to meet preset condition, execution step 203 'With step 203；In the case where determining to be unsatisfactory for preset condition, terminate flow.Wherein, preset condition can be pre-setVarious conditions.In general, if the sectional drawing of user interface meets preset condition, illustrate user interface there are the defects of it is smaller, fit at this timeConjunction is tested by testing the method for user interface；If the sectional drawing of user interface is unsatisfactory for preset condition, illustrate user interfaceThere are the defects of it is larger, do not fit through at this time test user interface method tested, therefore terminate flow.
In some optional realization methods of the present embodiment, preset condition can be sky, be defaulted as at this time to be measured on probationThe sectional drawing at family interface meets preset condition, and can directly perform step 203 ' and step 203.
In some optional realization methods of the present embodiment, preset condition can include but is not limited to following at least one：Character string matches unsuccessful, color category number more than preset number in default error character set of strings.
Specifically, first, electronic equipment can perform at least one of following operation：
1st, at least one of the sectional drawing of user interface to be tested character is identified using optical character identification OCR techniqueIt goes here and there and matches at least one character string identified in default error character set of strings.Here, OCR(Optical Character Recognition, optical character identification) technology refers to that electronic equipment checks cutting for user interfaceCharacter string in figure determines its shape by detecting dark, bright pattern, shape then is translated into calculating with character identifying methodThe technology of machine word.It, will be in the sectional drawing of user interface using optical mode i.e. for the character string in the sectional drawing of user interfaceCharacter string be converted into the image file of black and white lattice, and pass through identification software and the character string in image is converted into text latticeFormula, the technology further edited and processed for character processing software.Wherein, the character string in default error character set of strings can bePre-set various error character strings.For example, the character string in default error character set of strings can be user circle to be testedCharacter string in character string or mess code in the source code in face etc., if the character string identified is in default error character trailSuccessful match in conjunction then illustrates that the character string in source code is not converted completely into word to display in user interface to be testedThere are mess code on symbol string or user interface to be tested, illustrate at this time user interface there are the defects of it is larger, therefore terminate flow.
2nd, the color category included by the sectional drawing of user interface to be tested is determined using color model.Here, color modelCan be HSV (Hue, Saturation, Value) color model, hsv color model is also referred to as hexagonal pyramid model (HexconeModel), the parameter of color is respectively in this model：Tone (H), saturation degree (S), lightness (V).It can be fast using HSV modelsThe color category included by the sectional drawing of user interface to be tested is determined fastly.For example, if the sectional drawing of user interface to be tested is onlyIncluding a kind of color, then the sectional drawing of user interface to be tested is a pure color picture, illustrates do not have in user interface to be testedAny character.
Then, electronic equipment can be based at least one of above operation as a result, determining the sectional drawing of user interface to be testedWhether preset condition is met.
Specifically, if at least one of sectional drawing of user interface to be tested character string is in default error character set of stringsIt matches the color category included by the sectional drawing of user interface unsuccessful and/or to be tested and is more than preset number (such as a kind), then reallyThe sectional drawing of fixed user interface to be tested meets preset condition, then proceedes to perform step 203 ' and step 203；If user to be testedAt least one of the sectional drawing at interface character string successful match and/or user interface to be tested in default error character set of stringsSectional drawing included by color category be not more than preset number, it is determined that the sectional drawing of user interface to be tested is unsatisfactory for default itemThen part terminates flow.
Step 203 ', the sectional drawing of user interface to be tested is input to global test model, is obtained and user circle to be testedThe corresponding global test result in face.
In the present embodiment, in the case where the sectional drawing of user interface to be tested meets preset condition, electronic equipment can be withThe sectional drawing of user interface to be tested is input to global test model, so as to obtain global survey corresponding with user interface to be testedTest result.Wherein, global test result can only include global type, can also not only include global type but also including with the overall situationThe location information in the corresponding region of type.Global type can include but is not limited at least one of following：It is global normal type, completeOffice leaves blank type, global keyboard Exception Type.The overall situation leave blank type can be stayed including the overall situation black type and all be left white type.ExampleSuch as, for a user interface, global test result can be the location information in global leave blank type and region of leaving blank.
In the present embodiment, global test model can be used for characterizing the sectional drawing of user interface and corresponding with user interfaceCorrespondence between global test result.As an example, those skilled in the art can to the sectional drawing at a large number of users interface andGlobal test result corresponding with user interface is for statistical analysis, so as to generate be stored with multiple user interfaces sectional drawing and withThe mapping table of the corresponding global test result of user interface, and using the mapping table as global test model.ElectronicsEquipment can calculate the phase between the sectional drawing of user interface to be tested and the sectional drawing of each user interface in the mapping tableLike degree, and based on similarity calculation as a result, finding out global survey corresponding with user interface to be tested from the mapping tableTest result.For example, the sectional drawing with the highest user interface of the similarity of the sectional drawing of user interface to be tested is determined first, thenGlobal test corresponding with the sectional drawing of the user interface is found out from the mapping table as a result, as with user circle to be testedThe corresponding global test result in face.
In the present embodiment, global test model can be made of disaggregated model more than one, can also be by multiple single classificationModel forms.In the case where global test model is made of multiple single disaggregated models, global test model can include butIt is not limited at least one of following：The overall situation is left blank test model, global keyboard abnormality test model.
Step 203, at least one regional area is partitioned into from the sectional drawing of user interface to be tested.
In the present embodiment, in the case where the sectional drawing of user interface to be tested meets preset condition, electronic equipment can be withAt least one regional area is partitioned into from the sectional drawing of user interface to be tested.Wherein, each regional area includes a characterString.As an example, electronic equipment can identify at least one of the sectional drawing of user interface to be tested character using OCR techniqueString, is then partitioned into from the sectional drawing of user interface to be tested where each word at least one character string identifiedRegion.Wherein, the region where character string can be the minimum rectangular area for including character string.
Step 204, at least one regional area being partitioned into is input to local test model trained in advance, is obtainedLocal test result corresponding with user interface to be tested.
In the present embodiment, at least one regional area being partitioned into based on step 203, electronic equipment can will divideEach regional area at least one regional area cut out is input to local test model one by one, so as to obtain with it is to be testedThe corresponding local test result of user interface.Wherein, local test result can only include local type, can also both includeLocal type includes the location information in region corresponding with local type again.Local type can include but is not limited to it is following at leastOne：Character normal type, character overlap type, character background transition Exception Type, character edge type of barrier.It is for example, rightA regional area for including character being partitioned into the sectional drawing from user interface, local test result can be charactersThe location information of overlapping type and the regional area.
In the present embodiment, local test type can be used for characterizing regional area and and the user of the sectional drawing of user interfaceCorrespondence between the corresponding local test result in interface.As an example, those skilled in the art can be to a large amount of partial zonesDomain and local test result corresponding with regional area are for statistical analysis, so as to generate be stored with multiple regional areas and with officeThe mapping table of the corresponding local test result in portion region, and using the mapping table as local test model.Electronics is setBetween each regional area in the standby regional area and the mapping table of sectional drawing that can calculate user interface to be testedSimilarity, and based on similarity calculation as a result, finding out the office with the sectional drawing of user interface to be tested from the mapping tableThe corresponding local test result in portion region.For example, the phase with the regional area of the sectional drawing of user interface to be tested is determined firstLike highest regional area is spent, local test corresponding with the regional area is then found out from the mapping table as a result,As local test result corresponding with user interface to be tested.
In the present embodiment, local test model can be made of disaggregated model more than one, can also be by multiple single classificationModel forms.In the case where local test model is made of multiple single disaggregated models, local test model can include butIt is not limited at least one of following：Character overlap test model, character background transition abnormality test model, character edge block testModel.
Method provided by the embodiments of the present application for testing user interface, by obtaining cutting for user interface to be testedFigure, in order to which whether the sectional drawing for determining user interface to be tested meets preset condition；In the case where determining to meet preset condition,The sectional drawing of user interface to be tested is input to global test model first, it is corresponding with user interface to be tested complete so as to obtainOffice's test result；Then at least one regional area is partitioned into from the sectional drawing of user interface to be tested, will be partitioned intoA few regional area is input to local test model trained in advance, so as to obtain part corresponding with user interface to be testedTest result.Using deep learning method, user interface is tested with reference to global test model and local test model, noTester's artificial judgment is needed, not only saves human cost, also improves the testing efficiency to user interface.
With further reference to Fig. 3, it illustrates another implementations of the method for being used to test user interface according to the applicationThe flow 300 of example.This is used for the flow 300 for testing the method for user interface, includes the following steps：
Step 301, the sectional drawing of user interface to be tested is obtained.
In the present embodiment, it is (such as shown in FIG. 1 for testing the electronic equipment of the method for user interface operation thereonServer 105) it can from terminal device, (such as terminal shown in FIG. 1 be set by wired connection mode or radio connectionStandby 101,102,103) obtain the sectional drawing of user interface to be tested.In general, terminal device can be the electronics for having the function of sectional drawingEquipment is equipped with various client applications thereon.Terminal device can show that client should during client application is runUser interface, at this time terminal device sectional drawing function can be utilized to intercept shown user interface.
Step 302, determine whether the sectional drawing of user interface to be tested meets preset condition.
In the present embodiment, the sectional drawing based on the user interface to be tested acquired in step 301, electronic equipment can determineWhether the sectional drawing of user interface to be tested meets preset condition, in the case where determining to meet preset condition, execution step 303 'With step 303；In the case where determining to be unsatisfactory for preset condition, terminate flow.Wherein, preset condition can be pre-setVarious conditions.
Step 303 ', the sectional drawing of user interface to be tested is input to global test model, is obtained and user circle to be testedThe corresponding global test result in face.
In the present embodiment, in the case where the sectional drawing of user interface to be tested meets preset condition, electronic equipment can be withThe sectional drawing of user interface to be tested is input to global test model, so as to obtain global survey corresponding with user interface to be testedTest result.Wherein, global test result can only include global type, can also not only include global type but also including with the overall situationThe location information in the corresponding region of type.Global type can include but is not limited at least one of following：It is global normal type, completeOffice leaves blank type, global keyboard Exception Type.The overall situation leave blank type can be stayed including the overall situation black type and all be left white type.
Step 303, the DOM Document Object Model DOM of user interface to be tested is obtained.
In the present embodiment, electronic equipment can be obtained by wired connection mode or radio connection from terminal deviceTake the DOM (Document Object Model, DOM Document Object Model) of user interface to be tested.Here, on a user interface,The object of user interface is organized in a tree structure, and the master pattern for representing object in user interface is known as usingThe DOM at family interface.In practice, the operating system of terminal device can be Android (a kind of freedom and opening based on LinuxThe operating system of source code) or IOS (a kind of Mobile operating system).In general, no matter the operating system of terminal device isTextView can be included in Android or IOS, the DOM of user interface to be tested, and (TextView is for showing character stringComponent) information, the location information at least one First partial region can be included in TextView information, wherein, Mei GeOnly include a character string in one regional area, First partial region can be the smallest rectangular area for including this character stringDomain.
Step 304, according to the instruction of the location information at least one First partial region, from cutting for user interface to be testedAt least one First partial region is partitioned into figure, as at least one regional area.
In the present embodiment, the DOM information based on the user interface to be tested acquired in step 303, electronic equipment can be withAt least one first is determined in the sectional drawing of user interface to be tested according to the location information at least one First partial regionThen regional area is partitioned at least one First partial region, as at least one in the sectional drawing of user interface to be testedRegional area.
It is to be measured if the operating system of terminal device is Android in some optional realization methods of the present embodimentIt tries in the DOM of user interface to be also possible to that WebView can be included (WebView i.e. webpage view is the control for showing webpage)Information.If for example, in the DOM of user interface to be tested there are class attributes be " class='Android.webkit.WebView' " and there are the nodes of child node, then in the DOM that can determine user interface to be testedFurther include WebView information.Wherein, the location information of at least one second regional area can be included in WebView information, oftenA second regional area includes multiple character strings, and the second regional area can be the smallest rectangular area for including this multiple character stringDomain.Electronic equipment can be according to the instruction of the location information of at least one second regional area, from cutting for user interface to be testedAt least one second regional area is partitioned into figure；For each second regional area at least one second regional area,Multiple character strings in second regional area are identified using OCR technique and are partitioned into institute from second regional areaThe region where each character string in the multiple character strings identified, as multiple regional areas.
Step 305, at least one regional area being partitioned into is input to local test model trained in advance, is obtainedLocal test result corresponding with user interface to be tested.
In the present embodiment, at least one regional area being partitioned into based on step 304, electronic equipment can will divideEach regional area at least one regional area cut out is input to local test model one by one, so as to obtain with it is to be testedThe corresponding local test result of user interface.Wherein, local test result can only include local type, can also both includeLocal type includes the location information in region corresponding with local type again.Local type can include but is not limited to it is following at leastOne：Character normal type, character overlap type, character background transition Exception Type, character edge type of barrier.
From figure 3, it can be seen that compared with the corresponding embodiments of Fig. 2, in the present embodiment for testing user interfaceThe flow 300 of method highlights the step of segmentation regional area.The scheme of the present embodiment description can be by to be tested as a result,The DOM of user interface is analyzed, and realizes the location information for quickly determining regional area, so as to improve segmentation partial zonesThe efficiency in domain.
With further reference to Fig. 4, it illustrates the realities for being used to train the method for global test model according to the applicationApply the flow 400 of example.This is used for the flow 400 for training the method for global test model, includes the following steps：
Step 401, the sectional drawing at the sample of users interface for belonging to global normal type is obtained and with belonging to global normal typeThe corresponding global test in sample of users interface as a result, as global positive sample.
In the present embodiment, for training electronic equipment (such as Fig. 1 institutes of the method for global test model operation thereonThe server 105 shown) sectional drawing at the sample of users interface for belonging to global normal type can be obtained and with belonging to global normal classThe corresponding global test in sample of users interface of type is as a result, and as global positive sample.Wherein, belong to global normal typeThe sectional drawing at sample of users interface be input information in global positive sample, the sample of users interface with belonging to global normal typeCorresponding global test is the result is that output information in global positive sample.In general, belong to sample of users circle of global normal typeThe sectional drawing in face is expected identical with reference to figure with sample of users interface.
Step 402, it obtains the sectional drawing at the sample of users interface for belonging to global type of leaving blank and leaves blank type with belonging to the overall situationThe corresponding global test in sample of users interface as a result, as the first global negative sample.
In the present embodiment, electronic equipment can obtain the sample of users interface for belonging to global type of leaving blank sectional drawing and withBelong to the corresponding global test in sample of users interface of global type of leaving blank as a result, and as the first global negative sample.ItsIn, the sectional drawing for belonging to the sample of users interface of global type of leaving blank is the input information in the first global negative sample, complete with belonging toOffice leave blank type the corresponding global test in sample of users interface the result is that output information in the first global negative sample.In general,The sectional drawing for belonging to the sample of users interface of global type of leaving blank can be sample of users interface occur the overall situation leave blank type the defects ofWhen sectional drawing or the sectional drawing that is generated on the basis of the sectional drawing at the sample of users interface for belonging to global normal type.
In some optional realization methods of the present embodiment, the first default textures can be covered in by electronic equipment to be belonged toOn the sectional drawing at the sample of users interface of global normal type, generation belongs to the sectional drawing at the sample of users interface of global type of leaving blank.Wherein, the first default textures can be the pure color textures of default size and pixel value (such as rgb value).For example, area is not less than/ 5th of the sectional drawing at sample of users interface, and pure color textures of the rgb value between 0-10 or 250-255.Wherein, it utilizesPure color textures of the rgb value between 0-10 are covered on the sectional drawing at the sample of users interface for belonging to global normal type, Ke YishengInto the sectional drawing for belonging to the global sample of users interface for staying black type；It is covered in using pure color textures of the rgb value between 250-255Belong on the sectional drawing at sample of users interface of global normal type, can generate and belong to the global sample of users interface for being left white typeSectional drawing.
Step 403, using deep learning method, based on global positive sample and the first global negative sample to the preset first volumeProduct neural network is trained, and is obtained the overall situation and is left blank test model.
In the present embodiment, electronic equipment can be by sample of users circle for belonging to global normal type in global positive sampleThe sectional drawing at the sample of users interface for belonging to global type of leaving blank in the sectional drawing in face and the first global negative sample respectively as input,The corresponding global test result in the sample of users interface with belonging to global normal type in global positive sample and first is globalThe corresponding global test result in the sample of users interface with belonging to global type of leaving blank in negative sample is respectively as corresponding defeatedGo out, preset first convolutional neural networks (such as ResNet) are trained, leave blank test model so as to obtain the overall situation.ItsIn, convolutional neural networks (Convolutional Neural Network, CNN) can be a kind of feedforward neural network, itArtificial neuron can respond the surrounding cells in a part of coverage area, have outstanding performance for large-scale image procossing.Here,It can be by some different small random numbers of the network parameter (for example, weighting parameter and offset parameter) of the first convolutional neural networksIt is initialized, and uses BP (Back Propagation, backpropagation) algorithms or SGD in the training process(Stochastic Gradient Descent, stochastic gradient descent) algorithm is joined to adjust the network of the first convolutional neural networksNumber.
Step 402' obtains the sectional drawing at the sample of users interface for belonging to global keyboard Exception Type and with belonging to global keyboardThe corresponding global test in sample of users interface of Exception Type is as a result, as the second global negative sample.
In the present embodiment, electronic equipment can obtain the sectional drawing at the sample of users interface for belonging to global keyboard Exception TypeGlobal test corresponding with the sample of users interface for belonging to global keyboard Exception Type is as a result, and global negative as secondSample.Wherein, the sectional drawing for belonging to the sample of users interface of global keyboard Exception Type is the input letter in the second global negative sampleBreath, the corresponding global test in sample of users interface with belonging to global keyboard Exception Type is the result is that in the second global negative sampleOutput information.Occur in general, the sectional drawing for belonging to the sample of users interface of global keyboard Exception Type can be sample of users interfaceSectional drawing during the defects of global keyboard Exception Type or in the sectional drawing at the sample of users interface for belonging to global normal typeOn the basis of the sectional drawing that is generated.
In some optional realization methods of the present embodiment, default keyboard exception textures can be covered in by electronic equipmentBelong on the sectional drawing at sample of users interface of global normal type, generation belongs to the sample of users interface of global keyboard Exception TypeSectional drawing.Wherein, the key in the sectional drawing at sample of users interface that keyboard exception textures can be covered in global normal type is presetAt disk area.
Step 403', using deep learning method, based on global positive sample and the second global negative sample to preset secondConvolutional neural networks are trained, and obtain global keyboard abnormality test model.
In the present embodiment, electronic equipment can be by sample of users circle for belonging to global normal type in global positive sampleThe sectional drawing in face and the sectional drawing at the sample of users interface for belonging to global keyboard Exception Type in the second global negative sample respectively asInput, by the corresponding global test result in the sample of users interface with belonging to global normal type in global positive sample and secondThe corresponding global test result in the sample of users interface with belonging to global keyboard Exception Type in global negative sample respectively asCorresponding output is trained preset second convolutional neural networks (such as ResNet), abnormal so as to obtain global keyboardTest model.Here it is possible to the network parameter of the second convolutional neural networks is initialized with some different small random numbers,And the network parameter of the second convolutional neural networks is adjusted using BP algorithm or SGD algorithms in the training process.
In the optional realization method of in the present embodiment some, electronic equipment can be to belonging to the sample of global normal typeThe sectional drawing of this user interface, belong to global type of leaving blank sample of users interface sectional drawing and belong to the sample of global normal typeThe sectional drawing of user interface carries out figure isomery processing, generates multiple global positive samples, multiple first global negative samples and multiple theTwo global negative samples.Here, electronic equipment figure rotation, figure can be utilized to stretch, the figures isomery processing side such as pantographyFormula handles the sectional drawing at sample of users interface, so as to extend the quantity of the sectional drawing at sample of users interface.
With further reference to Fig. 5, it illustrates a realities of the method for the local test model of training according to the applicationApply the flow 500 of example.This includes the following steps for the flow 500 for the method for training local test model：
Step 501, the sample regional area for belonging to character normal type and the sample with belonging to character normal type are obtainedThe corresponding local test of regional area is as a result, as local positive sample.
In the present embodiment, electronic equipment (such as Fig. 1 institutes for the method operation of the local test model of training thereonThe server 105 shown) the sample regional area for belonging to character normal type and the sample with belonging to character normal type can be obtainedThe corresponding local test of this regional area is as a result, and as local positive sample.Wherein, belong to the sample of character normal typeRegional area is the input information in local positive sample, and the sample regional area with belonging to character normal type is corresponding locally to be surveyedTest result is the output information in local positive sample.In general, belong to the sample regional area of character normal type and sample partThe expected reference in region is schemed identical.
Step 502, the sample regional area for belonging to character overlap type and the sample with belonging to character overlap type are obtainedThe corresponding local test of regional area is as a result, as First partial negative sample.
In the present embodiment, electronic equipment can obtain the sample regional area that belongs to character overlap type and with belonging to wordThe corresponding local test of sample regional area of overlapping type is accorded with as a result, and as First partial negative sample.Wherein, belong toThe sample regional area of character overlap type is the input information in First partial negative sample, the sample with belonging to character overlap typeThe corresponding local test of this regional area is the result is that output information in First partial negative sample.In general, belong to character overlap classThe sample regional area of type can be figure when the defects of character overlap type occurs in sample regional area or belong toThe figure generated on the basis of the sample regional area of local normal type.
In some optional realization methods of the present embodiment, preset characters textures can be covered in by electronic equipment to be belonged toOn the sample regional area of character normal type, generation belongs to the sample regional area of character overlap type.
Step 503, using deep learning method, preset third is rolled up based on local positive sample and First partial negative sampleProduct neural network is trained, and obtains character overlap test model.
In the present embodiment, electronic equipment can be by the sample partial zones for belonging to character normal type in local positive sampleThe sample regional area for belonging to character overlap type in domain and First partial negative sample is respectively as input, by local positive sampleIn the corresponding local test result of the sample regional area with belonging to character normal type and First partial negative sample inBelong to the corresponding local test result of sample regional area of character overlap type respectively as corresponding output, to presetThree convolutional neural networks (such as ResNet) are trained, so as to obtain character overlap test model.Here it is possible to third is rolled upThe network parameter of product neural network is initialized with some different small random numbers, and in the training process using BP algorithm orPerson SGD algorithms adjust the network parameter of third convolutional neural networks.
Step 502', acquisition belong to the sample regional area of character background transition Exception Type and with belonging to character background mistakeThe corresponding local test of sample regional area of Exception Type is crossed as a result, as the second local negative sample.
In the present embodiment, electronic equipment can obtain belong to character background transition Exception Type sample regional area andThe corresponding local test of sample regional area with belonging to character background transition Exception Type is as a result, and as the second partNegative sample.Wherein, the sample regional area for belonging to character background transition Exception Type is the input letter in the second local negative sampleBreath, the corresponding local test of sample regional area with belonging to character background transition Exception Type is the result is that the second local negative sampleIn output information.In general, the sample regional area for belonging to character background transition Exception Type can be that sample regional area goes outFigure during the defects of existing character background transition Exception Type or in the sample regional area for belonging to local normal typeOn the basis of the figure that is generated.
In some optional realization methods of the present embodiment, default figure layer can be covered in by electronic equipment belongs to characterOn the sample regional area of normal type, generation belongs to the sample regional area of character background transition Exception Type.Wherein, it presetsFigure layer can be the figure layer of default transparency.For example, transparency is 50% figure layer.
Step 503', using deep learning method, based on local positive sample and the second local negative sample to the preset 4thConvolutional neural networks are trained, and obtain character background transition abnormality test model.
In the present embodiment, electronic equipment can be by the sample partial zones for belonging to character normal type in local positive sampleThe sample regional area for belonging to character background transition Exception Type in domain and the second local negative sample is respectively as input, by officeThe corresponding local test result of the sample regional area with belonging to character normal type in portion's positive sample and the negative sample in the second partThe corresponding local test result of sample regional area with belonging to character background transition Exception Type in this is respectively as correspondingOutput, to preset Volume Four product neural network (such as ResNet) be trained, so as to obtain the transition of character background exceptionTest model.Here it is possible to the network parameter of Volume Four product neural network is initialized with some different small random numbers,And the network parameter of Volume Four product neural network is adjusted using BP algorithm or SGD algorithms in the training process.
Step 502 〃 obtains the sample regional area for belonging to character edge type of barrier and blocks class with belonging to character edgeThe corresponding local test of sample regional area of type is as a result, as third part negative sample.
In the present embodiment, electronic equipment can obtain the sample regional area that belongs to character edge type of barrier and with categoryIn character edge type of barrier the corresponding local test of sample regional area as a result, and as third part negative sample.Wherein, the sample regional area for belonging to character edge type of barrier is the input information in the negative sample of third part, with belonging to wordThe corresponding local test of sample regional area of edge type of barrier is accorded with the result is that output information in the negative sample of third part.It is logicalOften, the sample regional area for belonging to character edge type of barrier can be that sample regional area character edge type of barrier occursFigure during defect or the figure generated on the basis of the sample regional area for belonging to local normal type.
In some optional realization methods of the present embodiment, the second default textures can be covered in by electronic equipment to be belonged toOn the edge of the sample regional area of character normal type, generation belongs to the sample partial zones that character edge blocks test modelDomain.Wherein, the second default textures can be the pure color textures of default size and pixel value (such as rgb value), the second default texturesThe edge of the character string in sample regional area can be covered in.
Step, 503 〃, using deep learning method, based on local positive sample and third part negative sample to the preset 5thConvolutional neural networks are trained, and are obtained character edge and are blocked test model.
In the present embodiment, electronic equipment can be by the sample partial zones for belonging to character normal type in local positive sampleThe sample regional area for belonging to character edge type of barrier in domain and third part negative sample, will locally just respectively as inputIn the corresponding local test result of the sample regional area with belonging to character normal type in sample and third part negative sampleThe corresponding local test result of the sample regional area with belonging to character edge type of barrier respectively as corresponding output, it is rightPreset 5th convolutional neural networks (such as ResNet) are trained, and test model is blocked so as to obtain character edge.Here,The network parameter of 5th convolutional neural networks can be initialized, and in the training process with some different small random numbersThe network parameter of the 5th convolutional neural networks is adjusted using BP algorithm or SGD algorithms.
In the optional realization method of in the present embodiment some, electronic equipment can be to belonging to the sample of character normal typeThis regional area, the sample regional area for belonging to character overlap type, the sample part for belonging to character background transition Exception TypeThe sample regional area progress figure isomery processing of test model is blocked in region with character edge is belonged to, and generates the positive sample in multiple partsSheet, multiple First partial negative samples, multiple second local negative samples and multiple third parts negative sample.Here, electronic equipment canTo be rotated using figure, figure stretches, the figures isomery processing mode such as pantography handles sample regional area, so as toExtend the quantity of sample regional area.
With further reference to Fig. 6, as the realization to method shown in above-mentioned each figure, used this application provides one kind for testingOne embodiment of the device at family interface, the device embodiment is corresponding with embodiment of the method shown in Fig. 2, which specifically may be usedTo be applied in various electronic equipments.
As shown in fig. 6, the present embodiment can include for testing the device 600 of user interface：Acquiring unit 601, reallyOrder member 602 and test cell 603.Wherein, acquiring unit 601 are configured to obtain the sectional drawing of user interface to be tested；It determinesUnit 602 is configured to determine whether the sectional drawing of user interface to be tested meets preset condition；Test cell 603, is configured toIn response to determining to meet preset condition, the sectional drawing of user interface to be tested is input to global test model, obtain with it is to be testedThe corresponding global test result of user interface；And at least one partial zones are partitioned into from the sectional drawing of user interface to be testedAt least one regional area being partitioned into is input to local test model trained in advance, obtained and user to be tested by domainThe corresponding local test in interface is as a result, wherein, each regional area includes a character string.
In the present embodiment, for testing in the device 600 of user interface：Acquiring unit 601, determination unit 602 and surveyThe specific processing and its caused technique effect for trying unit 603 can be respectively with reference to step 201, the steps in 2 corresponding embodiment of figureRapid 202 and step 203 and the related description of step 203'-204', details are not described herein.
In some optional realization methods of the present embodiment, determination unit 602 can include：Execution module is (in figure notShow), it is configured to carry out at least one of following operation：User circle to be tested is identified using optical character identification OCR techniqueAt least one of the sectional drawing in face character string and by least one character string identified in default error character set of stringsIn matched；The color category included by the sectional drawing of user interface to be tested is determined using color model；Determining module is (in figureBe not shown), be configured to it is based at least one of above operation as a result, determine user interface to be tested sectional drawing whether meet it is pre-If condition, wherein, preset condition includes at least one of following：Character string matched in default error character set of strings it is unsuccessful,The number of color category is more than preset number.
In some optional realization methods of the present embodiment, test cell 603 can include：DOM Document Object Model obtainsModule (not shown) is configured to obtain the DOM Document Object Model DOM of user interface to be tested, wherein, user to be testedIt can include the location information at least one First partial region in the DOM at interface, wherein, each First partial region includesOne character string；First partial region segmentation module (not shown) is configured to according at least one First partial regionLocation information instruction, at least one First partial region is partitioned into from the sectional drawing of user interface to be tested, as at leastOne regional area.
In some optional realization methods of the present embodiment, it can also include at least in the DOM of user interface to be testedThe location information of one the second regional area, wherein, each second regional area includes multiple character strings；And test cell603 can also include：Second local region segmentation module (not shown), is configured to according at least one second partial zonesThe instruction of the location information in domain is partitioned at least one second regional area from the sectional drawing of user interface to be tested；Partial zonesRegional partition module (not shown) is configured to for each second regional area at least one second regional area,Multiple character strings in second regional area are identified using OCR technique and are partitioned into institute from second regional areaThe region where each character string in the multiple character strings identified, as multiple regional areas.
In some optional realization methods of the present embodiment, global test result can include global type and with the overall situationThe location information in the corresponding region of type, local test result can include local type and region corresponding with local typeLocation information.
In some optional realization methods of the present embodiment, global type can include but is not limited to following at least one：Global normal type, the overall situation are left blank type, global keyboard Exception Type, and local type includes at least one of following：Character is justNormal type, character overlap type, character background transition Exception Type, character edge type of barrier.
In some optional realization methods of the present embodiment, global test model can include but is not limited to it is following at leastOne：The overall situation is left blank test model, global keyboard abnormality test model, and local test model includes at least one of following：CharacterOverlap test model, character background transition abnormality test model, character edge block test model.
In some optional realization methods of the present embodiment, the device 600 for testing user interface can also includeGlobal test model training unit (not shown), global test module training unit can include：Global positive sample obtainsModule (not shown) is configured to obtain the sectional drawing at the sample of users interface for belonging to global normal type and with belonging to the overall situationThe corresponding global test in sample of users interface of normal type is as a result, as global positive sample；First global negative sample obtains mouldBlock (not shown) is configured to obtain the sectional drawing at the sample of users interface for belonging to global type of leaving blank and be stayed with belonging to the overall situationThe corresponding global test in sample of users interface of void type is as a result, as the first global negative sample；Second global negative sample obtainsModule (not shown) is configured to obtain the sectional drawing at the sample of users interface for belonging to global keyboard Exception Type and with belonging toThe corresponding global test in sample of users interface of global keyboard Exception Type is as a result, as the second global negative sample；The overall situation is left blankTest model training module (not shown) is configured to using deep learning method, complete based on global positive sample and firstOffice's negative sample is trained preset first convolutional neural networks, obtains the overall situation and leaves blank test model；Global keyboard is surveyed extremelyModel training module (not shown) is tried, is configured to using deep learning method, it is global based on global positive sample and secondNegative sample is trained preset second convolutional neural networks, obtains global keyboard abnormality test model.
In some optional realization methods of the present embodiment, the device 600 for testing user interface can also includeGlobal sample generation unit (not shown), global sample generation unit can include：First global negative sample generation module(not shown) is configured to the first default textures being covered in the sectional drawing at the sample of users interface for belonging to global normal typeOn, generation belongs to the sectional drawing at the sample of users interface of global type of leaving blank；Second global negative sample generation module (does not show in figureGo out), it is configured to default keyboard exception textures being covered on the sectional drawing at the sample of users interface for belonging to global normal type, it is rawInto the sectional drawing at the sample of users interface for belonging to global keyboard Exception Type.
In some optional realization methods of the present embodiment, global sample generation unit can further include：First figureIsomery processing module (not shown) is configured to the sectional drawing at the sample of users interface to belonging to global normal type, belongs toThe overall situation leave blank type sample of users interface sectional drawing and the sectional drawing at sample of users interface that belongs to global normal type carry out figureThe processing of shape isomery generates multiple global positive samples, multiple first global negative samples and multiple second global negative samples.
In some optional realization methods of the present embodiment, the device 600 for testing user interface can also includeLocal test model training unit (not shown), local test module training unit can include：Local positive sample obtainsModule (not shown) is configured to acquisition and belongs to the sample regional area of character normal type and with belonging to the normal class of characterThe corresponding local test of sample regional area of type is as a result, as local positive sample；First partial negative sample acquisition module is (in figureIt is not shown), it is configured to obtain the sample regional area for belonging to character overlap type and the sample with belonging to character overlap typeThe corresponding local test of regional area is as a result, as First partial negative sample；Second local negative sample acquisition module (does not show in figureGo out), it is configured to obtain and belongs to the sample regional area of character background transition Exception Type and different with belonging to character background transitionThe corresponding local test of sample regional area of normal type is as a result, as the second local negative sample；Third part negative sample obtainsModule (not shown) is configured to acquisition and belongs to the sample regional area of character edge type of barrier and with belonging to character sideThe corresponding local test of sample regional area of edge type of barrier is as a result, as third part negative sample；Character overlap tests mouldType training module (not shown) is configured to, using deep learning method, sample be born based on local positive sample and First partialThis is trained preset third convolutional neural networks, obtains character overlap test model；Character background transition abnormality testModel training module (not shown) is configured to using deep learning method, negative based on local positive sample and the second partSample is trained preset Volume Four product neural network, obtains character background transition abnormality test model；Character edge hidesTest model training module (not shown) is kept off, is configured to using deep learning method, based on local positive sample and thirdLocal negative sample is trained preset 5th convolutional neural networks, obtains character edge and blocks test model.
In some optional realization methods of the present embodiment, the device 600 for testing user interface can also includeFractional sample generation unit (not shown), fractional sample generation unit can include：First partial negative sample generation module(not shown) is configured to preset characters textures being covered on the sample regional area for belonging to character normal type, rawInto the sample regional area for belonging to character overlap type；Second local negative sample generation module (not shown), is configured toDefault figure layer will be covered on the sample regional area for belonging to character normal type, generation belongs to character background transition exception classThe sample regional area of type；Third part negative sample generation module (not shown), being configured to will be by the second default texturesIt is covered on the edge for the sample regional area for belonging to character normal type, generation belongs to the sample that character edge blocks test modelThis regional area.
In some optional realization methods of the present embodiment, fractional sample generation unit can also include：Second graphIsomery processing module (not shown), be configured to belong to character normal type sample regional area, belong to character weightThe sample regional area of folded type belongs to the sample regional area of character background transition Exception Type and belongs to character edge and blocksThe sample regional area of test model carries out figure isomery processing, generate multiple local positive samples, multiple First partial negative samples,Multiple second local negative samples and multiple third parts negative sample.
Below with reference to Fig. 7, it illustrates suitable for being used for realizing the computer system 700 of the electronic equipment of the embodiment of the present applicationStructure diagram.Electronic equipment shown in Fig. 7 is only an example, to the function of the embodiment of the present application and should not use modelShroud carrys out any restrictions.
As shown in fig. 7, computer system 700 includes central processing unit (CPU) 701, it can be read-only according to being stored inProgram in memory (ROM) 702 or be loaded into program in random access storage device (RAM) 703 from storage section 708 andPerform various appropriate actions and processing.In RAM 703, also it is stored with system 700 and operates required various programs and data.CPU 701, ROM 702 and RAM 703 are connected with each other by bus 704.Input/output (I/O) interface 705 is also connected to alwaysLine 704.
I/O interfaces 705 are connected to lower component：Importation 706 including keyboard, mouse etc.；It is penetrated including such as cathodeThe output par, c 707 of spool (CRT), liquid crystal display (LCD) etc. and loud speaker etc.；Storage section 708 including hard disk etc.；And the communications portion 709 of the network interface card including LAN card, modem etc..Communications portion 709 via such as becauseThe network of spy's net performs communication process.Driver 710 is also according to needing to be connected to I/O interfaces 705.Detachable media 711, such asDisk, CD, magneto-optic disk, semiconductor memory etc. are mounted on driver 710, as needed in order to be read from thereonComputer program be mounted into storage section 708 as needed.
Particularly, in accordance with an embodiment of the present disclosure, it may be implemented as computer above with reference to the process of flow chart descriptionSoftware program.For example, embodiment of the disclosure includes a kind of computer program product, including being carried on computer-readable mediumOn computer program, which includes for the program code of the method shown in execution flow chart.In such realityIt applies in example, which can be downloaded and installed from network by communications portion 709 and/or from detachable media711 are mounted.When the computer program is performed by central processing unit (CPU) 701, perform what is limited in the present processesAbove-mentioned function.It should be noted that computer-readable medium described herein can be computer-readable signal media orComputer-readable medium either the two arbitrarily combines.Computer-readable medium for example can be --- but it is unlimitedIn --- electricity, magnetic, optical, electromagnetic, infrared ray or semiconductor system, device or device or it is arbitrary more than combination.It calculatesThe more specific example of machine readable medium can include but is not limited to：Being electrically connected with one or more conducting wires, portable meterCalculation machine disk, hard disk, random access storage device (RAM), read-only memory (ROM), erasable programmable read only memory(EPROM or flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), light storage device, magnetic memory device orThe above-mentioned any appropriate combination of person.In this application, computer-readable medium can any include or store having for programShape medium, the program can be commanded the either device use or in connection of execution system, device.And in the applicationIn, computer-readable signal media can be included in a base band or as the data-signal that a carrier wave part is propagated, whereinCarry computer-readable program code.Diversified forms may be used in the data-signal of this propagation, including but not limited to electricMagnetic signal, optical signal or above-mentioned any appropriate combination.Computer-readable signal media can also be computer-readable JieAny computer-readable medium other than matter, the computer-readable medium can send, propagate or transmit to be held by instructionRow system, device either device use or program in connection.The program code included on computer-readable mediumIt can be transmitted with any appropriate medium, including but not limited to：Wirelessly, electric wire, optical cable, RF etc. or above-mentioned arbitrary conjunctionSuitable combination.
Can with one or more programming language or combinations come write for perform the application operation calculatingMachine program code, described program design language include object-oriented programming language-such as Java, Smalltalk, C++, further include conventional procedural programming language-such as " C " language or similar programming language.Program code canFully to perform on the user computer, partly perform, performed as an independent software package on the user computer,Part performs or performs on a remote computer or server completely on the remote computer on the user computer for part.In situations involving remote computers, remote computer can pass through the network of any kind --- including LAN (LAN)Or wide area network (WAN)-be connected to subscriber computer or, it may be connected to outer computer (such as utilizes Internet serviceProvider passes through Internet connection).
Flow chart and block diagram in attached drawing, it is illustrated that according to the system of the various embodiments of the application, method and computer journeyArchitectural framework in the cards, function and the operation of sequence product.In this regard, each box in flow chart or block diagram can generationThe part of one module of table, program segment or code, the part of the module, program segment or code include one or more useIn the executable instruction of logic function as defined in realization.It should also be noted that it in some implementations as replacements, is marked in boxThe function of note can also be occurred with being different from the sequence marked in attached drawing.For example, two boxes succeedingly represented are actuallyIt can perform substantially in parallel, they can also be performed in the opposite order sometimes, this is depended on the functions involved.Also it to noteMeaning, the combination of each box in block diagram and/or flow chart and the box in block diagram and/or flow chart can be with holdingThe dedicated hardware based system of functions or operations as defined in row is realized or can use specialized hardware and computer instructionCombination realize.
Being described in unit involved in the embodiment of the present application can be realized by way of software, can also be by hardThe mode of part is realized.Described unit can also be set in the processor, for example, can be described as：A kind of processor packetInclude acquiring unit, determination unit and test cell.Wherein, the title of these units is not formed under certain conditions to the unitThe restriction of itself, for example, acquiring unit is also described as " unit for obtaining the sectional drawing of user interface to be tested ".
As on the other hand, present invention also provides a kind of computer-readable medium, which can beIncluded in electronic equipment described in above-described embodiment；Can also be individualism, and without be incorporated the electronic equipment in.Above computer readable medium carries one or more program, when said one or multiple programs are held by the electronic equipmentDuring row so that the electronic equipment：Obtain the sectional drawing of user interface to be tested；Determine whether the sectional drawing of user interface to be tested meetsPreset condition；In response to determining to meet preset condition, the sectional drawing of user interface to be tested is input to global test model, is obtainedGlobal test result corresponding with user interface to be tested；And it is partitioned into from the sectional drawing of user interface to be tested at least oneAt least one regional area being partitioned into is input in advance trained local test model by regional area, obtain with it is to be measuredThe corresponding local test of user interface is tried as a result, wherein, each regional area includes a character string.
The preferred embodiment and the explanation to institute's application technology principle that above description is only the application.People in the artMember should be appreciated that invention scope involved in the application, however it is not limited to the technology that the specific combination of above-mentioned technical characteristic formsScheme, while should also cover in the case where not departing from foregoing invention design, it is carried out by above-mentioned technical characteristic or its equivalent featureThe other technical solutions for arbitrarily combining and being formed.Such as features described above has similar work(with (but not limited to) disclosed hereinThe technical solution that the technical characteristic of energy is replaced mutually and formed.