Commit603440e

Leif Arne Storset

committed

Replace invalid characters with U+FFFD (fixeshtml5lib#96)

1 parentf5fd711 commit603440eCopy full SHA for 603440e

File tree

-0

lines changed

-0

lines changed

Lines changed: 1 addition & 0 deletions

Lines changed: 2 additions & 0 deletions

Original file line number	Diff line number	Diff line change
`@@ -7,6 +7,8 @@ Change Log`
`7`	`7`	`Released on XXX, 2014`
`8`	`8`
`9`	`9`	`* XXX`
	`10`	`+* Fix #96: replace invalid characters from "Preprocessing the input stream" with`
	`11`	`+ U+FFFD, preventing problems in lxml.`
`10`	`12`
`11`	`13`
`12`	`14`	`0.999`

Lines changed: 1 addition & 0 deletions

Original file line number	Diff line number	Diff line change
`@@ -270,6 +270,7 @@ def readChunk(self, chunkSize=None):`
`270`	`270`	`# Replace invalid characters`
`271`	`271`	`# Note U+0000 is dealt with in the tokenizer`
`272`	`272`	`data=self.replaceCharactersRegexp.sub("\ufffd",data)`
	`273`	`+data=invalid_unicode_re.sub("\ufffd",data)`
`273`	`274`
`274`	`275`	`data=data.replace("\r\n","\n")`
`275`	`276`	`data=data.replace("\r","\n")`

Comments

(0)