Commit30ea6e4

committed

Get rid of obsolete replacement of unpaired surrogates with U+FFFD.

1 parent7cce65b commit30ea6e4Copy full SHA for 30ea6e4

File tree

-5

lines changed

-5

lines changed

Lines changed: 0 additions & 5 deletions

Original file line number	Diff line number	Diff line change
`@@ -167,10 +167,8 @@ def __init__(self, source):`
`167`	`167`	`# Craziness`
`168`	`168`	`iflen("\U0010FFFF")==1:`
`169`	`169`	`self.reportCharacterErrors=self.characterErrorsUCS4`
`170`		`-self.replaceCharactersRegexp=re.compile("[\uD800-\uDFFF]")`
`171`	`170`	`else:`
`172`	`171`	`self.reportCharacterErrors=self.characterErrorsUCS2`
`173`		`-self.replaceCharactersRegexp=re.compile("([\uD800-\uDBFF](?![\uDC00-\uDFFF])\|(?<![\uD800-\uDBFF])[\uDC00-\uDFFF])")`
`174`	`172`
`175`	`173`	`# List of where new lines occur`
`176`	`174`	`self.newLines= [0]`
`@@ -268,9 +266,6 @@ def readChunk(self, chunkSize=None):`
`268`	`266`	`self.reportCharacterErrors(data)`
`269`	`267`
`270`	`268`	`# Replace invalid characters`
`271`		`-# Note U+0000 is dealt with in the tokenizer`
`272`		`-data=self.replaceCharactersRegexp.sub("\ufffd",data)`
`273`		`-`
`274`	`269`	`data=data.replace("\r\n","\n")`
`275`	`270`	`data=data.replace("\r","\n")`
`276`	`271`

Comments

(0)