io.TextIOWrapper and codecs.StreamReaderWriter offer the same API[1]. TextIOWrapper has more features and is faster thanStreamReaderWriter. Duplicate code means that bugs should be fixedtwice and that we may have subtle differences between the twoimplementations.
The codecs module was introduced in Python 2.0 (see thePEP 100).The io module wasintroduced in Python 2.6 and 3.0 (see thePEP 3116),and reimplemented in C inPython 2.7 and 3.1.
Further exploration of the concepts covered in this PEP has been deferredfor lack of a current champion interested in promoting the goals of the PEPand collecting and incorporating feedback, and with sufficient availabletime to do so effectively.
When the Python I/O model was updated for 3.0, the concept of a“stream-with-known-encoding” was introduced in the form ofio.TextIOWrapper. As this class is critical to the performance oftext-based I/O in Python 3, this module has an optimised C versionwhich is used by CPython by default. Many corner cases in handlingbuffering, stateful codecs and universal newlines have been dealt withsince the release of Python 3.0.
This new interface overlaps heavily with the legacycodecs.StreamReader, codecs.StreamWriter and codecs.StreamReaderWriterinterfaces that were part of the original codec interface design inPEP 100. These interfaces are organised around the principle of anencoding with an associated stream (i.e. the reverse of arrangement inthe io module), so the originalPEP 100 design required that codecwriters provide appropriate StreamReader and StreamWriterimplementations in addition to the core codec encode() and decode()methods. This places a heavy burden on codec authors providing thesespecialised implementations to correctly handle many of the cornercases (seeAppendix A) that have now been dealt with by io.TextIOWrapper. While deeperintegration between the codec and the stream allows for additionaloptimisations in theory, these optimisations have in practice eithernot been carried out and else the associated code duplication meansthat the corner cases that have been fixed in io.TextIOWrapper arestill not handled correctly in the various StreamReader andStreamWriter implementations.
Accordingly, this PEP proposes that:
Issues in the bug tracker:
By adding codec state read/write functions to the StreamReader andStreamWriter classes, it will become possible to fix issues withstateful codecs in a base class instead of in each statefulStreamReader and StreamWriter classes.
It would be possible to change StreamReader and StreamWriter to makethem use IncrementalDecoder and IncrementalEncoder.
A codec can implement variants which are optimized for the specificencoding or intercept certain stream methods to add functionality orimprove the encoding/decoding performance. TextIOWrapper cannotimplement such optimization, but TextIOWrapper uses incrementalencoders and decoders and uses read and write buffers, so the overheadof incomplete inputs is low or nul.
A lot more could be done for other variable length encoding codecs,e.g. UTF-8, since these often have problems near the end of a read dueto missing bytes. The UTF-32-BE/LE codecs could simply multiply thecharacter position by 4 to get the byte position.
These classes are rarely used directly, but indirectly usingcodecs.open(). They are not used in Python 3 standard library (exceptin the codecs module).
Some projects implement their own codec with StreamReader andStreamWriter, but don’t use these classes.
codecs.open() can be replaced by the builtin open() function. open()has a similar API but has also more options. Both functions returnfile-like objects (same API).
codecs.open() was the only way to open a text file in Unicode modeuntil Python 2.6. Many Python 2 programs uses this function. Removingcodecs.open() implies more work to port programs from Python 2 toPython 3, especially projects using the same code base for the twoPython versions (without using 2to3 program).
codecs.open() is kept for backward compatibility with Python 2.
Instantiating StreamReader or StreamWriter must emit a DeprecationWarning inPython 3.3. Defining a subclass doesn’t emit a DeprecationWarning.
codecs.open() will be changed to reuse the builtin open() function(TextIOWrapper) to read-write text files.
An alternative to the deprecation of the codecs.Stream* classes is to renamecodecs.open() to codecs.open_stream(), and to create a new codecs.open()function reusing open() and so io.TextIOWrapper.
It is difficult to use correctly a stateful codec with a stream. Somecases are supported by the codecs module, while io has no more knownbug related to stateful codecs. The main difference between the codecsand the io module is that bugs have to be fixed in StreamReader and/orStreamWriter classes of each codec for the codecs module, whereas bugscan be fixed only once in io.TextIOWrapper. Here are some examples ofissues with stateful codecs.
Python supports the following stateful codecs:
withopen(filename,'w',encoding='utf-16')asf:f.write('abc')f.write('def')f.seek(0)assertf.read()=='abcdef'f.seek(0)assertf.read()=='abcdef'
The io and codecs modules support this usecase correctly.
withopen(filename,'w',encoding='utf-16')asf:f.write('abc')pos=f.tell()withopen(filename,'w',encoding='utf-16')asf:f.seek(pos)f.write('def')f.seek(0)f.write('###')withopen(filename,'r',encoding='utf-16')asf:assertf.read()=='###def'
The io module supports this usecase, whereas codecs fails because itwrites a new BOM on the second write (issue #12512).
withopen(filename,'w',encoding='utf-16')asf:f.write('abc')withopen(filename,'a',encoding='utf-16')asf:f.write('def')withopen(filename,'r',encoding='utf-16')asf:assertf.read()=='abcdef'
The io module supports this usecase, whereas codecs fails because itwrites a new BOM on the second write (issue #12512).
This document has been placed in the public domain.
Source:https://github.com/python/peps/blob/main/peps/pep-0400.rst
Last modified:2025-02-01 08:59:27 GMT