Abstract

This PEP discusses a way in which Python can support I/O on fileswhich have a newline format that is not the native format on theplatform, so that Python on each platform can read and importfiles with CR (Macintosh), LF (Unix) or CR LF (Windows) lineendings.

It is more and more common to come across files that have an endof line that does not match the standard on the current platform:files downloaded over the net, remotely mounted filesystems on adifferent platform, Mac OS X with its double standard of Mac andUnix line endings, etc.

Many tools such as editors and compilers already handle thisgracefully, it would be good if Python did so too.

Specification

Universal newline support is enabled by default,but can be disabled during the configure of Python.

In a Python with universal newline support the feature isautomatically enabled for all import statements andexecfile()calls. There is no special support foreval() or exec.

In a Python with universal newline supportopen() the modeparameter can also be “U”, meaning “open for input as a text filewith universal newline interpretation”. Mode “rU” is also allowed,for symmetry with “rb”. Mode “U” cannot becombined with other mode flags such as “+”. Any line ending in theinput file will be seen as a'\n' in Python, so little other code hasto change to handle universal newlines.

Conversion of newlines happens in all calls that read data:read(),readline(),readlines(), etc.

There is no special support for output to file with a differentnewline convention, and so mode “wU” is also illegal.

A file object that has been opened in universal newline mode getsa new attribute “newlines” which reflects the newline conventionused in the file. The value for this attribute is one of None (nonewline read yet),"\r","\n","\r\n" or a tuple containing all thenewline types seen.

Rationale

Universal newline support is implemented in C, not in Python.This is done because we want files with a foreign newlineconvention to be import-able, so a Python Lib directory can beshared over a remote file system connection, or between MacPythonand Unix-Python on Mac OS X. For this to be feasible theuniversal newline convention needs to have a reasonably smallimpact on performance, which means a Python implementation is notan option as it would bog down all imports. And because of fileswith multiple newline conventions, which Visual C++ and otherWindows tools will happily produce, doing a quick check for thenewlines used in a file (handing off the import to C code if aplatform-local newline is seen) will not work. Finally, a Cimplementation also allows tracebacks and such (which open thePython source module) to be handled easily.

There is no output implementation of universal newlines, Pythonprograms are expected to handle this by themselves or write fileswith platform-local convention otherwise. The reason for this isthat input is the difficult case, outputting different newlines toa file is already easy enough in Python.

Also, an output implementation would be much more difficult than aninput implementation, surprisingly: a lot of output is done throughPyXXX_Print() methods, and at this point the file object is notavailable anymore, only aFILE*. So, an output implementation wouldneed to somehow go from theFILE* to the file object, because thatis where the current newline delimiter is stored.

The input implementation has no such problem: there are no cases inthe Python source tree where files are partially read from C,partially from Python, and such cases are expected to be rare inextension modules. If such cases exist the only problem is that thenewlines attribute of the file object is not updated during thefread() orfgets() calls that are done direct from C.

A partial output implementation, where strings passed tofp.write()would be converted to usefp.newlines as their line terminator butall other output would not is far too surprising, in my view.

Because there is no output support for universal newlines there isalso no support for a mode “rU+”: the surprise factor of theprevious paragraph would hold to an even stronger degree.

There is no support for universal newlines in strings passed toeval() orexec. It is envisioned that such strings always have thestandard\n line feed, if the strings come from a file that file canbe read with universal newlines.

I think there are no special issues with unicode. utf-16 shouldn’tpose any new problems, as such files need to be opened in binarymode anyway. Interaction with utf-8 is fine too: values 0x0a and 0x0dcannot occur as part of a multibyte sequence.

Universal newline files should work fine with iterators andxreadlines() as these eventually call the normal filereadline/readlines methods.

While universal newlines are automatically enabled for import theyare not for opening, where you have to specifically sayopen(...,"U"). This is open to debate, but here are a few reasons for thisdesign:

• Compatibility. Programs which already do their owninterpretation of\r\n in text files would break. Examples of suchprograms would be editors which warn you when you open a file witha different newline convention. If universal newlines was made thedefault such an editor would silently convert your line endings tothe local convention on save. Programs which open binary files astext files on Unix would also break (but it could be argued theydeserve it :-).
• Interface clarity. Universal newlines are only supported forinput files, not for input/output files, as the semantics wouldbecome muddy. Would you write Mac newlines if all reads so farhad encountered Mac newlines? But what if you then later read aUnix newline?

Thenewlines attribute is included so that programs that reallycare about the newline convention, such as text editors, canexamine what was in a file. They can then save (a copy of) thefile with the same newline convention (or, in case of a file withmixed newlines, ask the user what to do, or output in platformconvention).

Feedback is explicitly solicited on one item in the referenceimplementation: whether or not the universal newlines routinesshould grab the global interpreter lock. Currently they do not,but this could be considered living dangerously, as they maymodify fields in aFileObject. But as these routines arereplacements forfgets() andfread() as well it may be difficultto decide whether or not the lock is held when the routine iscalled. Moreover, the only danger is that if two threads read thesameFileObject at the same time an extraneous newline may be seenor thenewlines attribute may inadvertently be set to mixed. Iwould argue that if you read the sameFileObject in two threadssimultaneously you are asking for trouble anyway.

Note that no globally accessible pointers are manipulated in thefgets() orfread() replacement routines, just some integer-valuedflags, so the chances of core dumps are zero (he said:-).

Universal newline support can be disabled during configure because it doeshave a small performance penalty, and moreover the implementation hasnot been tested on all conceivable platforms yet. It might also be sillyon some platforms (WinCE or Palm devices, for instance). If universalnewline support is not enabled then file objects do not have thenewlinesattribute, so testing whether the current Python has it can be done with asimple:

ifhasattr(open,'newlines'):print'We have universal newline support'

Note that this test uses theopen() function rather than thefiletype so that it won’t fail for versions of Python where thefiletype was not available (thefile type was added to the built-innamespace in the same release as the universal newline feature wasadded).

Additionally, note that this test fails again on Python versions>= 2.5, whenopen() was made a function again and is not synonymouswith thefile type anymore.

Reference Implementation

A reference implementation is available in SourceForge patch#476814:https://bugs.python.org/issue476814

References

None.

Copyright

This document has been placed in the public domain.