Commitdda96f8

committed

Get rid of obsolete replacement of unpaired surrogates with U+FFFD.

1 parent46dae3d commitdda96f8Copy full SHA for dda96f8

File tree

-8

lines changed

-8

lines changed

-228 Bytes

Binary file not shown.

Lines changed: 1 addition & 8 deletions

Original file line number	Diff line number	Diff line change
`@@ -183,14 +183,10 @@ def __init__(self, source):`
`183`	`183`	`# Such platforms will have already checked for such`
`184`	`184`	`# surrogate errors, so no need to do this checking.`
`185`	`185`	`self.reportCharacterErrors=None`
`186`		`-self.replaceCharactersRegexp=None`
`187`	`186`	`eliflen("\U0010FFFF")==1:`
`188`	`187`	`self.reportCharacterErrors=self.characterErrorsUCS4`
`189`		`-self.replaceCharactersRegexp=re.compile(eval('"[\\uD800-\\uDFFF]"'))`
`190`	`188`	`else:`
`191`	`189`	`self.reportCharacterErrors=self.characterErrorsUCS2`
`192`		`-self.replaceCharactersRegexp=re.compile(`
`193`		`-eval('"([\\uD800-\\uDBFF](?![\\uDC00-\\uDFFF])\|(?<![\\uD800-\\uDBFF])[\\uDC00-\\uDFFF])"'))`
`194`	`190`
`195`	`191`	`# List of where new lines occur`
`196`	`192`	`self.newLines= [0]`
`@@ -288,10 +284,7 @@ def readChunk(self, chunkSize=None):`
`288`	`284`	`ifself.reportCharacterErrors:`
`289`	`285`	`self.reportCharacterErrors(data)`
`290`	`286`
`291`		`-# Replace invalid characters`
`292`		`-# Note U+0000 is dealt with in the tokenizer`
`293`		`-data=self.replaceCharactersRegexp.sub("\ufffd",data)`
`294`		`-`
	`287`	`+# Replace invalid characters`
`295`	`288`	`data=data.replace("\r\n","\n")`
`296`	`289`	`data=data.replace("\r","\n")`
`297`	`290`

Comments

(0)