Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commitfc9da96

Browse files
hartworkgpshead
andauthored
[3.11]gh-115398: Expose Expat >=2.6.0 reparse deferral API (CVE-2023-52425) (GH-115623) (#116268)
Allow controlling Expat >=2.6.0 reparse deferral (CVE-2023-52425) by adding five new methods:- `xml.etree.ElementTree.XMLParser.flush`- `xml.etree.ElementTree.XMLPullParser.flush`- `xml.parsers.expat.xmlparser.GetReparseDeferralEnabled`- `xml.parsers.expat.xmlparser.SetReparseDeferralEnabled`- `xml.sax.expatreader.ExpatParser.flush`Based on the "flush" idea from#115138 (comment) .- Please treat as a security fix related toCVE-2023-52425.(cherry picked from commit 6a95676)(cherry picked from commit73807eb)(cherry picked from commiteda2963)---------Includes code suggested-by: Snild Dolkow <snild@sony.com>and by core dev Serhiy Storchaka.Co-authored-by: Gregory P. Smith <greg@krypto.org>
1 parente8fb762 commitfc9da96

File tree

14 files changed

+435
-19
lines changed

14 files changed

+435
-19
lines changed

‎Doc/library/pyexpat.rst

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -196,6 +196,42 @@ XMLParser Objects
196196
:exc:`ExpatError` to be raised with the:attr:`code` attribute set to
197197
``errors.codes[errors.XML_ERROR_CANT_CHANGE_FEATURE_ONCE_PARSING]``.
198198

199+
..method::xmlparser.SetReparseDeferralEnabled(enabled)
200+
201+
..warning::
202+
203+
Calling ``SetReparseDeferralEnabled(False)`` has security implications,
204+
as detailed below; please make sure to understand these consequences
205+
prior to using the ``SetReparseDeferralEnabled`` method.
206+
207+
Expat 2.6.0 introduced a security mechanism called "reparse deferral"
208+
where instead of causing denial of service through quadratic runtime
209+
from reparsing large tokens, reparsing of unfinished tokens is now delayed
210+
by default until a sufficient amount of input is reached.
211+
Due to this delay, registered handlers may — depending of the sizing of
212+
input chunks pushed to Expat — no longer be called right after pushing new
213+
input to the parser. Where immediate feedback and taking over responsiblity
214+
of protecting against denial of service from large tokens are both wanted,
215+
calling ``SetReparseDeferralEnabled(False)`` disables reparse deferral
216+
for the current Expat parser instance, temporarily or altogether.
217+
Calling ``SetReparseDeferralEnabled(True)`` allows re-enabling reparse
218+
deferral.
219+
220+
Note that:meth:`SetReparseDeferralEnabled` has been backported to some
221+
prior releases of CPython as a security fix. Check for availability of
222+
:meth:`SetReparseDeferralEnabled` using:func:`hasattr` if used in code
223+
running across a variety of Python versions.
224+
225+
..versionadded::3.11.9
226+
227+
..method::xmlparser.GetReparseDeferralEnabled()
228+
229+
Returns whether reparse deferral is currently enabled for the given
230+
Expat parser instance.
231+
232+
..versionadded::3.11.9
233+
234+
199235
:class:`xmlparser` objects have the following attributes:
200236

201237

‎Doc/library/xml.etree.elementtree.rst

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -166,6 +166,11 @@ data but would still like to have incremental parsing capabilities, take a look
166166
at:func:`iterparse`. It can be useful when you're reading a large XML document
167167
and don't want to hold it wholly in memory.
168168

169+
Where *immediate* feedback through events is wanted, calling method
170+
:meth:`XMLPullParser.flush` can help reduce delay;
171+
please make sure to study the related security notes.
172+
173+
169174
Finding interesting elements
170175
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
171176

@@ -1378,6 +1383,24 @@ XMLParser Objects
13781383

13791384
Feeds data to the parser. *data* is encoded data.
13801385

1386+
1387+
..method::flush()
1388+
1389+
Triggers parsing of any previously fed unparsed data, which can be
1390+
used to ensure more immediate feedback, in particular with Expat >=2.6.0.
1391+
The implementation of:meth:`flush` temporarily disables reparse deferral
1392+
with Expat (if currently enabled) and triggers a reparse.
1393+
Disabling reparse deferral has security consequences; please see
1394+
:meth:`xml.parsers.expat.xmlparser.SetReparseDeferralEnabled` for details.
1395+
1396+
Note that:meth:`flush` has been backported to some prior releases of
1397+
CPython as a security fix. Check for availability of:meth:`flush`
1398+
using:func:`hasattr` if used in code running across a variety of Python
1399+
versions.
1400+
1401+
..versionadded::3.11.9
1402+
1403+
13811404
:meth:`XMLParser.feed` calls *target*\'s ``start(tag, attrs_dict)`` method
13821405
for each opening tag, its ``end(tag)`` method for each closing tag, and data
13831406
is processed by method ``data(data)``. For further supported callback
@@ -1439,6 +1462,22 @@ XMLPullParser Objects
14391462

14401463
Feed the given bytes data to the parser.
14411464

1465+
..method::flush()
1466+
1467+
Triggers parsing of any previously fed unparsed data, which can be
1468+
used to ensure more immediate feedback, in particular with Expat >=2.6.0.
1469+
The implementation of:meth:`flush` temporarily disables reparse deferral
1470+
with Expat (if currently enabled) and triggers a reparse.
1471+
Disabling reparse deferral has security consequences; please see
1472+
:meth:`xml.parsers.expat.xmlparser.SetReparseDeferralEnabled` for details.
1473+
1474+
Note that:meth:`flush` has been backported to some prior releases of
1475+
CPython as a security fix. Check for availability of:meth:`flush`
1476+
using:func:`hasattr` if used in code running across a variety of Python
1477+
versions.
1478+
1479+
..versionadded::3.11.9
1480+
14421481
..method::close()
14431482

14441483
Signal the parser that the data stream is terminated. Unlike

‎Include/pyexpat.h

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -48,8 +48,10 @@ struct PyExpat_CAPI
4848
enumXML_Status (*SetEncoding)(XML_Parserparser,constXML_Char*encoding);
4949
int (*DefaultUnknownEncodingHandler)(
5050
void*encodingHandlerData,constXML_Char*name,XML_Encoding*info);
51-
/* might benone for expat < 2.1.0 */
51+
/* might beNULL for expat < 2.1.0 */
5252
int (*SetHashSalt)(XML_Parserparser,unsigned longhash_salt);
53+
/* might be NULL for expat < 2.6.0 */
54+
XML_Bool (*SetReparseDeferralEnabled)(XML_Parserparser,XML_Boolenabled);
5355
/* always add new stuff to the end! */
5456
};
5557

‎Lib/test/test_pyexpat.py

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -758,5 +758,59 @@ def resolve_entity(context, base, system_id, public_id):
758758
self.assertEqual(handler_call_args, [("bar","baz")])
759759

760760

761+
classReparseDeferralTest(unittest.TestCase):
762+
deftest_getter_setter_round_trip(self):
763+
parser=expat.ParserCreate()
764+
enabled= (expat.version_info>= (2,6,0))
765+
766+
self.assertIs(parser.GetReparseDeferralEnabled(),enabled)
767+
parser.SetReparseDeferralEnabled(False)
768+
self.assertIs(parser.GetReparseDeferralEnabled(),False)
769+
parser.SetReparseDeferralEnabled(True)
770+
self.assertIs(parser.GetReparseDeferralEnabled(),enabled)
771+
772+
deftest_reparse_deferral_enabled(self):
773+
ifexpat.version_info< (2,6,0):
774+
self.skipTest(f'Expat{expat.version_info} does not '
775+
'support reparse deferral')
776+
777+
started= []
778+
779+
defstart_element(name,_):
780+
started.append(name)
781+
782+
parser=expat.ParserCreate()
783+
parser.StartElementHandler=start_element
784+
self.assertTrue(parser.GetReparseDeferralEnabled())
785+
786+
forchunkin (b'<doc',b'/>'):
787+
parser.Parse(chunk,False)
788+
789+
# The key test: Have handlers already fired? Expecting: no.
790+
self.assertEqual(started, [])
791+
792+
parser.Parse(b'',True)
793+
794+
self.assertEqual(started, ['doc'])
795+
796+
deftest_reparse_deferral_disabled(self):
797+
started= []
798+
799+
defstart_element(name,_):
800+
started.append(name)
801+
802+
parser=expat.ParserCreate()
803+
parser.StartElementHandler=start_element
804+
ifexpat.version_info>= (2,6,0):
805+
parser.SetReparseDeferralEnabled(False)
806+
self.assertFalse(parser.GetReparseDeferralEnabled())
807+
808+
forchunkin (b'<doc',b'/>'):
809+
parser.Parse(chunk,False)
810+
811+
# The key test: Have handlers already fired? Expecting: yes.
812+
self.assertEqual(started, ['doc'])
813+
814+
761815
if__name__=="__main__":
762816
unittest.main()

‎Lib/test/test_sax.py

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@
1919
fromioimportBytesIO,StringIO
2020
importcodecs
2121
importos.path
22+
importpyexpat
2223
importshutil
2324
importsys
2425
fromurllib.errorimportURLError
@@ -1214,6 +1215,56 @@ def test_expat_incremental_reset(self):
12141215

12151216
self.assertEqual(result.getvalue(),start+b"<doc>text</doc>")
12161217

1218+
deftest_flush_reparse_deferral_enabled(self):
1219+
ifpyexpat.version_info< (2,6,0):
1220+
self.skipTest(f'Expat{pyexpat.version_info} does not support reparse deferral')
1221+
1222+
result=BytesIO()
1223+
xmlgen=XMLGenerator(result)
1224+
parser=create_parser()
1225+
parser.setContentHandler(xmlgen)
1226+
1227+
forchunkin ("<doc",">"):
1228+
parser.feed(chunk)
1229+
1230+
self.assertEqual(result.getvalue(),start)# i.e. no elements started
1231+
self.assertTrue(parser._parser.GetReparseDeferralEnabled())
1232+
1233+
parser.flush()
1234+
1235+
self.assertTrue(parser._parser.GetReparseDeferralEnabled())
1236+
self.assertEqual(result.getvalue(),start+b"<doc>")
1237+
1238+
parser.feed("</doc>")
1239+
parser.close()
1240+
1241+
self.assertEqual(result.getvalue(),start+b"<doc></doc>")
1242+
1243+
deftest_flush_reparse_deferral_disabled(self):
1244+
result=BytesIO()
1245+
xmlgen=XMLGenerator(result)
1246+
parser=create_parser()
1247+
parser.setContentHandler(xmlgen)
1248+
1249+
forchunkin ("<doc",">"):
1250+
parser.feed(chunk)
1251+
1252+
ifpyexpat.version_info>= (2,6,0):
1253+
parser._parser.SetReparseDeferralEnabled(False)
1254+
1255+
self.assertEqual(result.getvalue(),start)# i.e. no elements started
1256+
self.assertFalse(parser._parser.GetReparseDeferralEnabled())
1257+
1258+
parser.flush()
1259+
1260+
self.assertFalse(parser._parser.GetReparseDeferralEnabled())
1261+
self.assertEqual(result.getvalue(),start+b"<doc>")
1262+
1263+
parser.feed("</doc>")
1264+
parser.close()
1265+
1266+
self.assertEqual(result.getvalue(),start+b"<doc></doc>")
1267+
12171268
# ===== Locator support
12181269

12191270
deftest_expat_locator_noinfo(self):

‎Lib/test/test_xml_etree.py

Lines changed: 63 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -121,10 +121,6 @@
121121
</foo>
122122
"""
123123

124-
fails_with_expat_2_6_0= (unittest.expectedFailure
125-
ifpyexpat.version_info>= (2,6,0)else
126-
lambdatest:test)
127-
128124
defcheckwarnings(*filters,quiet=False):
129125
defdecorator(test):
130126
defnewtest(*args,**kwargs):
@@ -1382,12 +1378,14 @@ def test_attlist_default(self):
13821378

13831379
classXMLPullParserTest(unittest.TestCase):
13841380

1385-
def_feed(self,parser,data,chunk_size=None):
1381+
def_feed(self,parser,data,chunk_size=None,flush=False):
13861382
ifchunk_sizeisNone:
13871383
parser.feed(data)
13881384
else:
13891385
foriinrange(0,len(data),chunk_size):
13901386
parser.feed(data[i:i+chunk_size])
1387+
ifflush:
1388+
parser.flush()
13911389

13921390
defassert_events(self,parser,expected,max_events=None):
13931391
self.assertEqual(
@@ -1405,34 +1403,32 @@ def assert_event_tags(self, parser, expected, max_events=None):
14051403
self.assertEqual([(action,elem.tag)foraction,eleminevents],
14061404
expected)
14071405

1408-
deftest_simple_xml(self,chunk_size=None):
1406+
deftest_simple_xml(self,chunk_size=None,flush=False):
14091407
parser=ET.XMLPullParser()
14101408
self.assert_event_tags(parser, [])
1411-
self._feed(parser,"<!-- comment -->\n",chunk_size)
1409+
self._feed(parser,"<!-- comment -->\n",chunk_size,flush)
14121410
self.assert_event_tags(parser, [])
14131411
self._feed(parser,
14141412
"<root>\n <element key='value'>text</element",
1415-
chunk_size)
1413+
chunk_size,flush)
14161414
self.assert_event_tags(parser, [])
1417-
self._feed(parser,">\n",chunk_size)
1415+
self._feed(parser,">\n",chunk_size,flush)
14181416
self.assert_event_tags(parser, [('end','element')])
1419-
self._feed(parser,"<element>text</element>tail\n",chunk_size)
1420-
self._feed(parser,"<empty-element/>\n",chunk_size)
1417+
self._feed(parser,"<element>text</element>tail\n",chunk_size,flush)
1418+
self._feed(parser,"<empty-element/>\n",chunk_size,flush)
14211419
self.assert_event_tags(parser, [
14221420
('end','element'),
14231421
('end','empty-element'),
14241422
])
1425-
self._feed(parser,"</root>\n",chunk_size)
1423+
self._feed(parser,"</root>\n",chunk_size,flush)
14261424
self.assert_event_tags(parser, [('end','root')])
14271425
self.assertIsNone(parser.close())
14281426

1429-
@fails_with_expat_2_6_0
14301427
deftest_simple_xml_chunk_1(self):
1431-
self.test_simple_xml(chunk_size=1)
1428+
self.test_simple_xml(chunk_size=1,flush=True)
14321429

1433-
@fails_with_expat_2_6_0
14341430
deftest_simple_xml_chunk_5(self):
1435-
self.test_simple_xml(chunk_size=5)
1431+
self.test_simple_xml(chunk_size=5,flush=True)
14361432

14371433
deftest_simple_xml_chunk_22(self):
14381434
self.test_simple_xml(chunk_size=22)
@@ -1631,6 +1627,57 @@ def test_unknown_event(self):
16311627
withself.assertRaises(ValueError):
16321628
ET.XMLPullParser(events=('start','end','bogus'))
16331629

1630+
deftest_flush_reparse_deferral_enabled(self):
1631+
ifpyexpat.version_info< (2,6,0):
1632+
self.skipTest(f'Expat{pyexpat.version_info} does not '
1633+
'support reparse deferral')
1634+
1635+
parser=ET.XMLPullParser(events=('start','end'))
1636+
1637+
forchunkin ("<doc",">"):
1638+
parser.feed(chunk)
1639+
1640+
self.assert_event_tags(parser, [])# i.e. no elements started
1641+
ifETispyET:
1642+
self.assertTrue(parser._parser._parser.GetReparseDeferralEnabled())
1643+
1644+
parser.flush()
1645+
1646+
self.assert_event_tags(parser, [('start','doc')])
1647+
ifETispyET:
1648+
self.assertTrue(parser._parser._parser.GetReparseDeferralEnabled())
1649+
1650+
parser.feed("</doc>")
1651+
parser.close()
1652+
1653+
self.assert_event_tags(parser, [('end','doc')])
1654+
1655+
deftest_flush_reparse_deferral_disabled(self):
1656+
parser=ET.XMLPullParser(events=('start','end'))
1657+
1658+
forchunkin ("<doc",">"):
1659+
parser.feed(chunk)
1660+
1661+
ifpyexpat.version_info>= (2,6,0):
1662+
ifnotETispyET:
1663+
self.skipTest(f'XMLParser.(Get|Set)ReparseDeferralEnabled '
1664+
'methods not available in C')
1665+
parser._parser._parser.SetReparseDeferralEnabled(False)
1666+
1667+
self.assert_event_tags(parser, [])# i.e. no elements started
1668+
ifETispyET:
1669+
self.assertFalse(parser._parser._parser.GetReparseDeferralEnabled())
1670+
1671+
parser.flush()
1672+
1673+
self.assert_event_tags(parser, [('start','doc')])
1674+
ifETispyET:
1675+
self.assertFalse(parser._parser._parser.GetReparseDeferralEnabled())
1676+
1677+
parser.feed("</doc>")
1678+
parser.close()
1679+
1680+
self.assert_event_tags(parser, [('end','doc')])
16341681

16351682
#
16361683
# xinclude tests (samples from appendix C of the xinclude specification)

‎Lib/xml/etree/ElementTree.py

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1328,6 +1328,11 @@ def read_events(self):
13281328
else:
13291329
yieldevent
13301330

1331+
defflush(self):
1332+
ifself._parserisNone:
1333+
raiseValueError("flush() called after end of stream")
1334+
self._parser.flush()
1335+
13311336

13321337
defXML(text,parser=None):
13331338
"""Parse XML document from string constant.
@@ -1734,6 +1739,15 @@ def close(self):
17341739
delself.parser,self._parser
17351740
delself.target,self._target
17361741

1742+
defflush(self):
1743+
was_enabled=self.parser.GetReparseDeferralEnabled()
1744+
try:
1745+
self.parser.SetReparseDeferralEnabled(False)
1746+
self.parser.Parse(b"",False)
1747+
exceptself._errorasv:
1748+
self._raiseerror(v)
1749+
finally:
1750+
self.parser.SetReparseDeferralEnabled(was_enabled)
17371751

17381752
# --------------------------------------------------------------------
17391753
# C14N 2.0

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp