Commit542dc08

James Graham

committed

Move tests that are problematic in json or java to a single file. Add double escaping flag to sneak unpaired surrogates past over-zealous json decoders

1 parent51460c0 commit542dc08Copy full SHA for 542dc08

File tree

1 file changed

+21

-2

lines changed

html5lib/tests
- test_tokenizer.py

1 file changed

+21

-2

lines changed

`‎html5lib/tests/test_tokenizer.py`

Lines changed: 21 additions & 2 deletions

Original file line number	Diff line number	Diff line change
`@@ -86,7 +86,7 @@ def normalizeTokens(tokens):`
`86`	`86`	`fori,tokeninenumerate(tokens):`
`87`	`87`	`iftoken[0]==u'ParseError':`
`88`	`88`	`tokens[i]=token[0]`
`89`		`-returnjson.loads(json.dumps(tokens))`
	`89`	`+returntokens`
`90`	`90`
`91`	`91`	`deftokensMatch(expectedTokens,receivedTokens,ignoreErrorOrder):`
`92`	`92`	`"""Test whether the test has passed or failed`
`@@ -121,12 +121,31 @@ def tokensMatch(expectedTokens, receivedTokens, ignoreErrorOrder):`
`121`	`121`
`122`	`122`	`returntokens["expected"]==tokens["received"]`
`123`	`123`
	`124`	`+defunescape_test(test):`
	`125`	`+defdecode(inp):`
	`126`	`+returninp.decode("unicode-escape")`
	`127`	`+`
	`128`	`+test["input"]=decode(test["input"])`
	`129`	`+fortokenintest["output"]:`
	`130`	`+iftoken=="ParseError":`
	`131`	`+continue`
	`132`	`+else:`
	`133`	`+token[1]=decode(token[1])`
	`134`	`+iflen(token)>2:`
	`135`	`+forkey,valueintoken[2]:`
	`136`	`+deltoken[2][key]`
	`137`	`+token[2][decode(key)]=decode(value)`
	`138`	`+sys.stderr.write(str(test))`
	`139`	`+returntest`
`124`	`140`
`125`	`141`	`classTestCase(unittest.TestCase):`
`126`	`142`	`defrunTokenizerTest(self,test):`
`127`	`143`	`#XXX - move this out into the setup function`
`128`	`144`	`#concatenate all consecutive character tokens into a single token`
`129`		`-expected=concatenateCharacterTokens(test['output'])`
	`145`	`+if'doubleEscaped'intest:`
	`146`	`+test=unescape_test(test)`
	`147`	`+`
	`148`	`+expected=concatenateCharacterTokens(test['output'])`
`130`	`149`	`if'lastStartTag'notintest:`
`131`	`150`	`test['lastStartTag']=None`
`132`	`151`	`outBuffer=cStringIO.StringIO()`

0 commit comments

Comments

(0)

Movatterモバイル変換

Navigation Menu

Search code, repositories, users, issues, pull requests...

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Commit542dc08

File tree

1 file changed

1 file changed

`‎html5lib/tests/test_tokenizer.py`

0 commit comments