| Kent Tamura | 59ffb02 | 2018-11-27 05:30:56 | [diff] [blame] | 1 | # How to Extend the Web Test Framework |
| andybons | 3322f76 | 2015-08-24 21:37:09 | [diff] [blame] | 2 | |
| Kent Tamura | 59ffb02 | 2018-11-27 05:30:56 | [diff] [blame] | 3 | TheWebTestFramework thatBlink usesis a regression testing tool thatis |
| andybons | ad92aa3 | 2015-08-31 02:27:44 | [diff] [blame] | 4 | multi-platformand it has a large amount of tools that help test varying types |
| 5 | of regression, suchas pixel diffs, text diffs, etc.The frameworkis mainly |
| 6 | usedbyBlink, however it was made to be extensible so that other projects can |
| 7 | use it test different parts of chrome(suchasPrintPreview).Thisis a guide |
| 8 | to help people who want to actually the framework to test whatever they want. |
| andybons | 3322f76 | 2015-08-24 21:37:09 | [diff] [blame] | 9 | |
| andybons | ad92aa3 | 2015-08-31 02:27:44 | [diff] [blame] | 10 | [TOC] |
| 11 | |
| 12 | ## Background |
| 13 | |
| 14 | Before you can start actually extending the framework, you should be familiar |
| Kent Tamura | 59ffb02 | 2018-11-27 05:30:56 | [diff] [blame] | 15 | with how touse it.See the[web tests documentation](testing/web_tests.md). |
| andybons | 3322f76 | 2015-08-24 21:37:09 | [diff] [blame] | 16 | |
| andybons | ad92aa3 | 2015-08-31 02:27:44 | [diff] [blame] | 17 | ## How to Extend the Framework |
| 18 | |
| andybons | 3322f76 | 2015-08-24 21:37:09 | [diff] [blame] | 19 | There are two parts to actually extending framework to test a piece of software. |
| 20 | The first partis extending certain filesin: |
| Kent Tamura | 0101944 | 2018-05-01 22:06:58 | [diff] [blame] | 21 | [/third_party/blink/tools/blinkpy/web_tests/](/third_party/blink/tools/blinkpy/web_tests/) |
| Kent Tamura | 59ffb02 | 2018-11-27 05:30:56 | [diff] [blame] | 22 | The codein`blinkpy/web_tests`is the web test framework itself |
| andybons | 3322f76 | 2015-08-24 21:37:09 | [diff] [blame] | 23 | |
| andybons | ad92aa3 | 2015-08-31 02:27:44 | [diff] [blame] | 24 | The second partis creating a driver(program) to actually communicate the |
| Kent Tamura | 59ffb02 | 2018-11-27 05:30:56 | [diff] [blame] | 25 | web test framework.This partis significantly more trickyand dependent on |
| andybons | ad92aa3 | 2015-08-31 02:27:44 | [diff] [blame] | 26 | what exactly exactlyis being tested. |
| andybons | 3322f76 | 2015-08-24 21:37:09 | [diff] [blame] | 27 | |
| andybons | ad92aa3 | 2015-08-31 02:27:44 | [diff] [blame] | 28 | ### Part 1 |
| andybons | 3322f76 | 2015-08-24 21:37:09 | [diff] [blame] | 29 | |
| andybons | ad92aa3 | 2015-08-31 02:27:44 | [diff] [blame] | 30 | This part isn’t too difficult.There are basically two classes that need to be |
| 31 | extended(ideally, just inheritedfrom).These classes are: |
| andybons | 3322f76 | 2015-08-24 21:37:09 | [diff] [blame] | 32 | |
| Kent Tamura | 59ffb02 | 2018-11-27 05:30:56 | [diff] [blame] | 33 | *`Driver`.Locatedin`web_tests/port/driver.py`.Each instance ofthisis |
| nodir | a6074d4c | 2015-09-01 04:26:45 | [diff] [blame] | 34 | theclass that will actually an instance of the program that produces the |
| 35 | test data(programinPart2). |
| Kent Tamura | 59ffb02 | 2018-11-27 05:30:56 | [diff] [blame] | 36 | *`Port`.Locatedin`web_tests/port/base.py`.Thisclassis responsible |
| nodir | a6074d4c | 2015-09-01 04:26:45 | [diff] [blame] | 37 | creating driverswith the correct settings, giving access to certain OS |
| 38 | functionality to access expected files, etc. |
| andybons | 3322f76 | 2015-08-24 21:37:09 | [diff] [blame] | 39 | |
| andybons | ad92aa3 | 2015-08-31 02:27:44 | [diff] [blame] | 40 | #### Extending Driver |
| andybons | 3322f76 | 2015-08-24 21:37:09 | [diff] [blame] | 41 | |
| andybons | ad92aa3 | 2015-08-31 02:27:44 | [diff] [blame] | 42 | As said,Driver launches the programfromPart2.Said program will communicate |
| 43 | with the driverclass to receive instructionsand send back data.All of the |
| 44 | workfor driver getsdonein`Driver.run_test`.Everythingelseis a helperor |
| 45 | initializationfunction. |
| andybons | 3322f76 | 2015-08-24 21:37:09 | [diff] [blame] | 46 | |
| andybons | ad92aa3 | 2015-08-31 02:27:44 | [diff] [blame] | 47 | `run_test()` steps: |
| 48 | |
| 49 | 1.On the very first call ofthisfunction, it will actually run the test |
| 50 | program.On every subsequent call tothisfunction, at the beginning it will |
| 51 | verify that the process doesn’t need to be restarted,andif it does, it |
| 52 | will create anew instance of the test program. |
| 53 | 1.It willthen create a command to send the program |
| 54 | *This command generally consists of an html file pathfor the test |
| 55 | program to navigate to. |
| 56 | *After creating it, the commandis sent |
| 57 | 1.After the command has been sent, it willthen waitfor datafrom the |
| 58 | program. |
| 59 | *It will actually waitfor2 blocks of data. |
| 60 | *The first part being textor audio data.This partis required(the |
| 61 | program will always send something, even an emptystring) |
| 62 | *The second blockis optionalandis image dataand an image hash |
| 63 | (md5)this block of datais usedfor pixel tests |
| 64 | 1.After it has received all the data, it will proceed to checkif the program |
| 65 | has timedoutor crashed,andif so failthis instance of the test(it can |
| 66 | be retried laterif need be). |
| 67 | |
| 68 | Luckily,`run_test()` most likely doesn’t need to be overriddenunless extra |
| 69 | blocks of data need to be sent to/readfrom the test program.However, youdo |
| 70 | need to know how it works because it will influence what functions you need to |
| 71 | override.Here are the ones you’re probably going to need tooverride |
| 72 | |
| 73 | cmd_line |
| 74 | |
| 75 | Thisfunction creates aset of command line arguments to run the test program, |
| 76 | so thefunction will almost certainly need to be overridden. |
| 77 | |
| 78 | It creates the command line to run the program.`Driver` uses`subprocess.popen` |
| 79 | to create the process, which takes the name of the test programand any options |
| 80 | it might need. |
| 81 | |
| 82 | The first itemin the list of arguments should be the path to test programusing |
| 83 | thisfunction: |
| 84 | |
| 85 | self._port._path_to_driver() |
| 86 | |
| 87 | Thisis an absolute path to the test program.Thisis the bare minimum you need |
| 88 | toget the driver to launch the test program, howeverif you have options you |
| 89 | need to append, just append them to the list. |
| 90 | |
| 91 | start |
| 92 | |
| 93 | If your program has any special startup needs,thenthis will be the place to |
| 94 | put it. |
| 95 | |
| 96 | That’s mostly it.TheDriverclass has almost all the functionality you could |
| 97 | want, so there isn’t much tooverride here.If extra data needs to be reador |
| 98 | sent, extra data members should be added to`ContentBlock`. |
| 99 | |
| 100 | #### Extending Port |
| 101 | |
| 102 | Thisclassis responsiblefor providing functionality suchaswhere to lookfor |
| 103 | tests,where to store test results, what driver to run, what timeout touse, |
| 104 | what kind of files can be run, etc.It provides a lot of functionality, however |
| 105 | it isn’t really sufficient because it doesn’t account of platform specific |
| 106 | problems, therefore port itself shouldn’t be extend.InsteadLinuxPort,WinPort, |
| 107 | andMacPort(and maybe the android portclass) should be extendedas they |
| 108 | provide platform specific overrides/extensions that implement most of the |
| 109 | important functionality.While there are many functionsinPort, overriding one |
| 110 | function will affect most of the other ones toget the desired behavior.For |
| Kent Tamura | 81f891f | 2018-12-12 01:36:15 | [diff] [blame] | 111 | example,if`web_tests_dir()`is overridden,not only will the code lookfor |
| andybons | ad92aa3 | 2015-08-31 02:27:44 | [diff] [blame] | 112 | testsin that directory, but it will find the correctTestExpectations file, the |
| 113 | platform specific expected files, etc. |
| andybons | 3322f76 | 2015-08-24 21:37:09 | [diff] [blame] | 114 | |
| 115 | Here are some of the functions that most likely need to be overridden. |
| nodir | a6074d4c | 2015-09-01 04:26:45 | [diff] [blame] | 116 | |
| andybons | ad92aa3 | 2015-08-31 02:27:44 | [diff] [blame] | 117 | *`driver_class` |
| 118 | *This should be overridden to allow the testing program to actually run. |
| 119 | Bydefault the code will run content_shell, which mightor mightnot be |
| 120 | what you want. |
| 121 | *It should be overridden toreturn the driver extensionclass created |
| 122 | earlier.Thisfunction doesn’treturn an instance on the driver, just |
| 123 | theclass itself. |
| 124 | *`driver_name` |
| 125 | *This shouldreturn the name of the program test p.Bydefault it returns |
| 126 | ‘content_shell’, but you want to have itreturn the program you want to |
| 127 | run, suchas`chrome`or`browser_tests`. |
| Kent Tamura | 81f891f | 2018-12-12 01:36:15 | [diff] [blame] | 128 | *`web_tests_dir` |
| andybons | ad92aa3 | 2015-08-31 02:27:44 | [diff] [blame] | 129 | *This tells the portwhere to lookfor all theand everything associated |
| 130 | with them suchas resources files. |
| Kent Tamura | 59ffb02 | 2018-11-27 05:30:56 | [diff] [blame] | 131 | *Bydefault it returns the absolute path to the web tests directory. |
| andybons | ad92aa3 | 2015-08-31 02:27:44 | [diff] [blame] | 132 | *If you are planning on running somethingin the chromium src/ directory, |
| 133 | there are helper functions to allow you toreturn a path relative to the |
| 134 | base of the chromium src directory. |
| andybons | 3322f76 | 2015-08-24 21:37:09 | [diff] [blame] | 135 | |
| andybons | ad92aa3 | 2015-08-31 02:27:44 | [diff] [blame] | 136 | The rest of the functions can definitely be overriddenfor your projects |
| 137 | specific needs, however these are the bare minimum needed toget it running. |
| 138 | There are also functions you canoverride to make certain actions that aren’t on |
| Kent Tamura | 59ffb02 | 2018-11-27 05:30:56 | [diff] [blame] | 139 | bydefault always take place.For example, the web test framework always |
| andybons | ad92aa3 | 2015-08-31 02:27:44 | [diff] [blame] | 140 | checksfor system dependenciesunless youpassin aswitch.If you want them |
| 141 | disabledfor your project, justoverride`check_sys_deps` to alwaysreturn OK. |
| 142 | This way you don’t need topassin so many switches. |
| andybons | 3322f76 | 2015-08-24 21:37:09 | [diff] [blame] | 143 | |
| andybons | ad92aa3 | 2015-08-31 02:27:44 | [diff] [blame] | 144 | As said earlier, you shouldoverrideLinuxPort,MacPort,and/orWinPort.You |
| 145 | should create aclass thatimplements the platform independent overrides(such |
| 146 | as`driver_class`)andthen create a separateclassfor each platform specific |
| 147 | port of your program that inheritsfrom theclasswith the independent overrides |
| 148 | and the platform port you want.For example, you might want to have a different |
| 149 | timeoutfor your project, but onWindows the timeout needs to be vastly |
| 150 | different than the others.Inthiscase you can just create adefaultoverride |
| 151 | that everyclass usesexcept yourWindows port.In that port you can just |
| 152 | override thefunction again to provide the specific timeout you need.This way |
| 153 | you don’t need to maintain the samefunction on each platformif they alldo the |
| 154 | same thing. |
| andybons | 3322f76 | 2015-08-24 21:37:09 | [diff] [blame] | 155 | |
| andybons | ad92aa3 | 2015-08-31 02:27:44 | [diff] [blame] | 156 | For`Driver`and`Port` that’s basically itunless you need to make many odd |
| 157 | modifications.Lots of functionalityis already there so you shouldn’t really |
| 158 | need todo much. |
| andybons | 3322f76 | 2015-08-24 21:37:09 | [diff] [blame] | 159 | |
| andybons | ad92aa3 | 2015-08-31 02:27:44 | [diff] [blame] | 160 | ### Part 2 |
| andybons | 3322f76 | 2015-08-24 21:37:09 | [diff] [blame] | 161 | |
| andybons | ad92aa3 | 2015-08-31 02:27:44 | [diff] [blame] | 162 | Thisis the partwhere you create the program that your driverclass launches. |
| 163 | This partis very application dependent, so it willnot be a guide on how |
| 164 | implement certain features, just what should be implementedand the orderin |
| 165 | which events should occurand some guidelines about what todo/notdo.For a |
| 166 | good example of how to implement your test program, look atMockDRTin |
| 167 | `mock_drt.pyin` the same directoryas`base.py`and`driver.py`.It goes through |
| 168 | all the steps described belowandis very clearand concise.Itis writtenin |
| 169 | python, but your driver can be anything that can be runby`subprocess.popen` |
| 170 | and has stdout, stdin, stderr. |
| andybons | 3322f76 | 2015-08-24 21:37:09 | [diff] [blame] | 171 | |
| andybons | ad92aa3 | 2015-08-31 02:27:44 | [diff] [blame] | 172 | #### Goals |
| 173 | |
| 174 | Your goalforthis part of the projectis to create a program(or extend a |
| Kent Tamura | 81f891f | 2018-12-12 01:36:15 | [diff] [blame] | 175 | program) tointerfacewith the web test framework.The web test framework |
| andybons | ad92aa3 | 2015-08-31 02:27:44 | [diff] [blame] | 176 | will communicatewiththis program to tell it what todoand it will accept data |
| 177 | fromthis program to perform the regression testingor createnewbase line |
| 178 | files. |
| 179 | |
| 180 | #### Structure |
| 181 | |
| andybons | 3322f76 | 2015-08-24 21:37:09 | [diff] [blame] | 182 | Thisis how your code should be laidout. |
| andybons | ad92aa3 | 2015-08-31 02:27:44 | [diff] [blame] | 183 | |
| 184 | 1.Initialization |
| 185 | *The creation of any directoriesor the launching of any programs should |
| 186 | bedone hereand should bedone once. |
| 187 | *After the programis initialized,“#READY\n” should be sent to progress |
| 188 | the`run_test()`in the driver. |
| 189 | 1.InfiniteLoop(!) |
| 190 | *After initialization, your program needs to actually waitfor input, |
| Kent Tamura | 59ffb02 | 2018-11-27 05:30:56 | [diff] [blame] | 191 | then process that input to carryout the test.In the context of web |
| andybons | ad92aa3 | 2015-08-31 02:27:44 | [diff] [blame] | 192 | testing, the`content_shell` needs to waitfor an html file to navigate |
| 193 | to, render it,then convert that rendering to a PNG.It doesthis |
| 194 | constantly,until a signal/messageis sent to indicate thatno more |
| 195 | tests should be processed |
| 196 | *Details: |
| 197 | *The first thing you needis your test file pathand any other |
| 198 | additional information about the test thatis required(thisis sent |
| 199 | during the write() stepin`run_tests()`is`driver.py`.This |
| 200 | information will be passed through stdinandis just one large |
| 201 | string,with each part of the command being splitwith apostrophes |
| 202 | (ex:“/path’foo”is path to the test file,then foois some setting |
| 203 | that your program might need). |
| 204 | *After that, your program should act onthis input, how it doesthis |
| 205 | is dependent on your program, howeverin`content_shell`,this would |
| 206 | be the partwhere it navigates to the test file,then renders it. |
| 207 | After the program acts on the input, it needs to send some text to |
| 208 | the driver code to indicate that it has acted on the input.This |
| 209 | text will indicate something that you want to test.For example,if |
| 210 | you want to make sure you program always prints“foo” you should |
| 211 | send it to the driver.If the program every prints“bar”(or |
| 212 | anythingelse), that would indicate a failureand the test will |
| 213 | fail. |
| 214 | *Then you need to send any image datain the same manneras you did |
| 215 | for step2. |
| 216 | *Cleanup everything related to processing the inputfrom step i,then |
| 217 | go back to step1. |
| 218 | *Thisiswhere the‘infinite’ loop part comesin, your program |
| 219 | should constantly accept inputfrom the driveruntil the driver |
| 220 | indicates that there areno more tests to run.The driver doesthis |
| 221 | by closing stdin, which will cause std::cin to gointo a bad state. |
| 222 | However, you can also modify the driver to send a specialstring |
| 223 | suchas‘QUIT’ toexit thewhile loop. |
| andybons | 3322f76 | 2015-08-24 21:37:09 | [diff] [blame] | 224 | |
| 225 | That’s basically what the skeleton of your program should be. |
| 226 | |
| andybons | ad92aa3 | 2015-08-31 02:27:44 | [diff] [blame] | 227 | ### Details |
| andybons | 3322f76 | 2015-08-24 21:37:09 | [diff] [blame] | 228 | |
| andybons | ad92aa3 | 2015-08-31 02:27:44 | [diff] [blame] | 229 | Thisis information about how todo some specific things, suchas sending data |
| Kent Tamura | 59ffb02 | 2018-11-27 05:30:56 | [diff] [blame] | 230 | to the web test framework. |
| andybons | 3322f76 | 2015-08-24 21:37:09 | [diff] [blame] | 231 | |
| andybons | ad92aa3 | 2015-08-31 02:27:44 | [diff] [blame] | 232 | *ContentBlocks |
| Kent Tamura | 59ffb02 | 2018-11-27 05:30:56 | [diff] [blame] | 233 | *The web test framework accepts outputfrom your programin blocks of |
| andybons | ad92aa3 | 2015-08-31 02:27:44 | [diff] [blame] | 234 | data through stdout.Therefore, printing to stdoutis really sending |
| Kent Tamura | 59ffb02 | 2018-11-27 05:30:56 | [diff] [blame] | 235 | data to the web test framework. |
| andybons | ad92aa3 | 2015-08-31 02:27:44 | [diff] [blame] | 236 | *Structure of block |
| 237 | *“Header:Data\n” |
| 238 | *Header indicates what type of data will be sent through. A list |
| 239 | of valid headersis listedin`Driver.py`. |
| 240 | *Datais the data that you actually want to send.For pixel |
| 241 | tests, you want to send the actual PNG data here. |
| 242 | *The newlineis needed to indicate theend of a header. |
| 243 | *End of a content block |
| 244 | *To indicate theend of a a content blockand cause the driver to |
| 245 | progress, you need to write“#EOF\n” to stdout(mandatory)and |
| 246 | to stderrfor certain types of content, suchas image data. |
| 247 | *Multiple headers per block |
| 248 | *Some blocksrequire different sets of data.ForPNGs,not only |
| 249 | is the PNG needed, but sois a hash of the bitmap used to create |
| 250 | the PNG. |
| 251 | *Inthiscasethisis how your output should look. |
| 252 | *“Content-type: image/png\n” |
| 253 | *“ActualHash: hashData\n” |
| 254 | *“Content-Length: lengthOfPng\n” |
| 255 | *“pngdata” |
| 256 | *This part doesn’t need a header specifying that you are |
| 257 | sending png data, just send it |
| 258 | *“#EOF\n” on both stdoutand stderr |
| 259 | *To see the structure of the data required, look at the |
| 260 | `read_block` functionsinDriver.py |
