Confusing set of OS-related exceptions

OS-related (or system call-related) exceptions are currently a diversityof classes, arranged in the following sub-hierarchies:

+--EnvironmentError+--IOError+--io.BlockingIOError+--io.UnsupportedOperation(alsoinheritsfromValueError)+--socket.error+--socket.gaierror+--socket.herror+--socket.timeout+--OSError+--VMSError+--WindowsError+--mmap.error+--select.error

While some of these distinctions can be explained by implementationconsiderations, they are often not very logical at a higher level. Theline separating OSError and IOError, for example, is often blurry. Considerthe following:

>>>os.remove("fff")Traceback (most recent call last):  File"<stdin>", line1, in<module>OSError:[Errno 2] No such file or directory: 'fff'>>>open("fff")Traceback (most recent call last):  File"<stdin>", line1, in<module>IOError:[Errno 2] No such file or directory: 'fff'

The same error condition (a non-existing file) gets cast as two differentexceptions depending on which library function was called. The reasonfor this is that theos module exclusively raises OSError (or itssubclass WindowsError) while theio module mostly raises IOError.However, the user is interested in the nature of the error, not in whichpart of the interpreter it comes from (since the latter is obvious fromreading the traceback message or application source code).

In fact, it is hard to think of any situation where OSError should becaught but not IOError, or the reverse.

A further proof of the ambiguity of this segmentation is that the standardlibrary itself sometimes has problems deciding. For example, in theselect module, similar failures will raiseselect.error,OSErrororIOError depending on whether you are using select(), a poll object,a kqueue object, or an epoll object. This makes user code uselesslycomplicated since it has to be prepared to catch various exception types,depending on which exact implementation of a single primitive it choosesto use at runtime.

As for WindowsError, it seems to be a pointless distinction. First, itonly exists on Windows systems, which requires tedious compatibility codein cross-platform applications (such code can be found inLib/shutil.py).Second, it inherits from OSError and is raised for similar errors as OSErroris raised for on other systems. Third, the user wanting access to low-levelexception specifics has to examine theerrno orwinerror attributeanyway.

Lack of fine-grained exceptions

The current variety of OS-related exceptions doesn’t allow the user to filtereasily for the desired kinds of failures. As an example, consider the taskof deleting a file if it exists. The Look Before You Leap (LBYL) idiomsuffers from an obvious race condition:

ifos.path.exists(filename):os.remove(filename)

If a file named asfilename is created by another thread or processbetween the calls toos.path.exists andos.remove, it won’t bedeleted. This can produce bugs in the application, or even security issues.

Therefore, the solution is to try to remove the file, and ignore the errorif the file doesn’t exist (an idiom known as Easier to Ask Forgivenessthan to get Permission, or EAFP). Careful code will read like the following(which works under both POSIX and Windows systems):

try:os.remove(filename)exceptOSErrorase:ife.errno!=errno.ENOENT:raise

or even:

try:os.remove(filename)exceptEnvironmentErrorase:ife.errno!=errno.ENOENT:raise

This is a lot more to type, and also forces the user to remember the variouscryptic mnemonics from theerrno module. It imposes an additionalcognitive burden and gets tiresome rather quickly. Consequently, manyprogrammers will instead write the following code, which silences exceptionstoo broadly:

try:os.remove(filename)exceptOSError:pass

os.remove can raise an OSError not only when the file doesn’t exist,but in other possible situations (for example, the filename points to adirectory, or the current process doesn’t have permission to removethe file), which all indicate bugs in the application logic and thereforeshouldn’t be silenced. What the programmer would like to write instead issomething such as:

try:os.remove(filename)exceptFileNotFoundError:pass

Justification

Not only does this first step present the user a simpler landscape asexplained in therationale section, but it also allows for a betterand more complete resolution ofStep 2 (seePrerequisite).

The rationale for keepingOSError as the official name for genericOS-related exceptions is that it, precisely, is more generic thanIOError.EnvironmentError is more tedious to type and also much lesser-known.

The survey inAppendix B shows that IOError is thedominant error today in the standard library. As for third-party Python code,Google Code Search shows IOError being ten times more popular thanEnvironmentError in user code, and three times more popular than OSError[3]. However, with no intention to deprecate IOError in the middleterm, the lesser popularity of OSError is not a problem.

Exception attributes

Since WindowsError is coalesced into OSError, the latter gains awinerrorattribute under Windows. It is set to None under situations where it is notmeaningful, as is already the case with theerrno,filename andstrerror attributes (for example when OSError is raised directly byPython code).

Deprecation of names

The following paragraphs outline a possible deprecation strategy forold exception names. However, it has been decided to keep them as aliasesfor the time being. This decision could be revised in time for Python 4.0.

Prerequisite

Step 1 is a loose prerequisite for this.

Prerequisite, because some errnos can currently be attached to differentexception classes: for example, ENOENT can be attached to both OSError andIOError, depending on the context. If we don’t want to breakusefulcompatibility, we can’t make anexceptOSError (or IOError) fail tomatch an exception where it would succeed today.

Loose, because we could decide for a partial resolution of step 2if existing exception classes are not coalesced: for example, ENOENT couldraise a hypothetical FileNotFoundError where an IOError was previouslyraised, but continue to raise OSError otherwise.

The dependency on step 1 could be totally removed if the new subclassesused multiple inheritance to match with all of the existing superclasses(or, at least, OSError and IOError, which are arguable the most prevalentones). It would, however, make the hierarchy more complicated andtherefore harder to grasp for the user.

New exception classes

The following tentative list of subclasses, along with a description andthe list of errnos mapped to them, is submitted to discussion:

• FileExistsError: trying to create a file or directory which alreadyexists (EEXIST)
• FileNotFoundError: for all circumstances where a file and directory isrequested but doesn’t exist (ENOENT)
• IsADirectoryError: file-level operation (open(), os.remove()…)requested on a directory (EISDIR)
• NotADirectoryError: directory-level operation requested on somethingelse (ENOTDIR)
• PermissionError: trying to run an operation without the adequate accessrights - for example filesystem permissions (EACCES, EPERM)
• BlockingIOError: an operation would block on an object (e.g. socket) setfor non-blocking operation (EAGAIN, EALREADY, EWOULDBLOCK, EINPROGRESS);this is the existingio.BlockingIOError with an extended role
• BrokenPipeError: trying to write on a pipe while the other end has beenclosed, or trying to write on a socket which has been shutdown for writing(EPIPE, ESHUTDOWN)
• InterruptedError: a system call was interrupted by an incoming signal(EINTR)
• ConnectionAbortedError: connection attempt aborted by peer (ECONNABORTED)
• ConnectionRefusedError: connection reset by peer (ECONNREFUSED)
• ConnectionResetError: connection reset by peer (ECONNRESET)
• TimeoutError: connection timed out (ETIMEDOUT); this can be re-castas a generic timeout exception, replacingsocket.timeout and also usefulfor other types of timeout (for example in Lock.acquire())
• ChildProcessError: operation on a child process failed (ECHILD);this is raised mainly by the wait() family of functions.
• ProcessLookupError: the given process (as identified by, e.g., itsprocess id) doesn’t exist (ESRCH).

In addition, the following exception class is proposed for inclusion:

• ConnectionError: a base class forConnectionAbortedError,ConnectionRefusedError andConnectionResetError

The following drawing tries to sum up the proposed additions, along withthe corresponding errno values (where applicable). The root of thesub-hierarchy (OSError, assumingStep 1 is accepted in full) is notshown:

+--BlockingIOErrorEAGAIN,EALREADY,EWOULDBLOCK,EINPROGRESS+--ChildProcessErrorECHILD+--ConnectionError+--BrokenPipeErrorEPIPE,ESHUTDOWN+--ConnectionAbortedErrorECONNABORTED+--ConnectionRefusedErrorECONNREFUSED+--ConnectionResetErrorECONNRESET+--FileExistsErrorEEXIST+--FileNotFoundErrorENOENT+--InterruptedErrorEINTR+--IsADirectoryErrorEISDIR+--NotADirectoryErrorENOTDIR+--PermissionErrorEACCES,EPERM+--ProcessLookupErrorESRCH+--TimeoutErrorETIMEDOUT

Naming

Various naming controversies can arise. One of them is whether allexception class names should end in “Error”. In favour is consistencywith the rest of the exception hierarchy, against is concision (especiallywith long names such asConnectionAbortedError).

Exception attributes

In order to preserveuseful compatibility, these subclasses should stillset adequate values for the various exception attributes defined on thesuperclass (for exampleerrno,filename, and optionallywinerror).

Implementation

Since it is proposed that the subclasses are raised based purely on thevalue oferrno, little or no changes should be required in extensionmodules (either standard or third-party).

The first possibility is to adapt thePyErr_SetFromErrno() familyof functions (PyErr_SetFromWindowsErr() under Windows) to raise theappropriate OSError subclass. This wouldn’t cover, however, Pythoncode raising OSError directly, using the following idiom (seen inLib/tempfile.py):

raiseIOError(_errno.EEXIST,"No usable temporary file name found")

A second possibility, suggested by Marc-Andre Lemburg, is to adaptOSError.__new__ to instantiate the appropriate subclass. This hasthe benefit of also covering Python code such as the above.

Namespace pollution

Making the exception hierarchy finer-grained makes the root (or builtins)namespace larger. This is to be moderated, however, as:

• only a handful of additional classes are proposed;
• while standard exception types live in the root namespace, they arevisually distinguished by the fact that they use the CamelCase convention,while almost all other builtins use lowercase naming (except True, False,None, Ellipsis and NotImplemented)

An alternative would be to provide a separate module containing thefiner-grained exceptions, but that would defeat the purpose ofencouraging careful code over careless code, since the user would firsthave to import the new module instead of using names already accessible.

Pattern matching

Another possibility would be to introduce an advanced pattern matchingsyntax when catching exceptions. For example:

try:os.remove(filename)exceptOSErroraseife.errno==errno.ENOENT:pass

Several problems with this proposal:

• it introduces new syntax, which is perceived by the author to be a heavierchange compared to reworking the exception hierarchy
• it doesn’t decrease typing effort significantly
• it doesn’t relieve the programmer from the burden of having to remembererrno mnemonics

Common errnos with OSError

• EBADF: bad file descriptor (usually means the file descriptor wasclosed)
• EEXIST: file or directory exists
• EINTR: interrupted function call
• EISDIR: is a directory
• ENOTDIR: not a directory
• ENOENT: no such file or directory
• EOPNOTSUPP: operation not supported on socket(possible confusion with the existing io.UnsupportedOperation)
• EPERM: operation not permitted (when using e.g. os.setuid())

Common errnos with IOError

• EACCES: permission denied (for filesystem operations)
• EBADF: bad file descriptor (with select.epoll); read operation on awrite-only GzipFile, or vice-versa
• EBUSY: device or resource busy
• EISDIR: is a directory (when trying to open())
• ENODEV: no such device
• ENOENT: no such file or directory (when trying to open())
• ETIMEDOUT: connection timed out

Common errnos with socket.error

All these errors may also be associated with a plain IOError, for examplewhen calling read() on a socket’s file descriptor.

• EAGAIN: resource temporarily unavailable (during a non-blocking socketcall except connect())
• EALREADY: connection already in progress (during a non-blockingconnect())
• EINPROGRESS: operation in progress (during a non-blocking connect())
• EINTR: interrupted function call
• EISCONN: the socket is connected
• ECONNABORTED: connection aborted by peer (during an accept() call)
• ECONNREFUSED: connection refused by peer
• ECONNRESET: connection reset by peer
• ENOTCONN: socket not connected
• ESHUTDOWN: cannot send after transport endpoint shutdown
• EWOULDBLOCK: same reasons asEAGAIN

Common errnos with select.error

• EINTR: interrupted function call

About VMSError

VMSError is completely unused by the interpreter core and the standardlibrary. It was added as part of the OpenVMS patches submitted in 2002by Jean-François Piéronne[4]; the motivation for including VMSError was thatit could be raised by third-party packages.

Interpreter core

Handling of PYTHONSTARTUP raises IOError (but the error gets discarded):

$ PYTHONSTARTUP=foox ./pythonPython 3.2a0 (py3k:82920M, Jul 16 2010, 22:53:23)[GCC 4.4.3] on linux2Type "help", "copyright", "credits" or "license" for more information.Could not open PYTHONSTARTUPIOError: [Errno 2] No such file or directory: 'foox'

PyObject_Print() raises IOError when ferror() signals an error on theFILE* parameter (which, in the source tree, is always either stdout orstderr).

Unicode encoding and decoding using thembcs encoding can raiseWindowsError for some error conditions.

Standard library

>>>bz2.BZ2File("foox","rb")Traceback (most recent call last):  File"<stdin>", line1, in<module>IOError:[Errno 2] No such file or directory>>>bz2.BZ2File("LICENSE","rb").read()Traceback (most recent call last):  File"<stdin>", line1, in<module>IOError:invalid data stream>>>bz2.BZ2File("/tmp/zzz.bz2","wb").read()Traceback (most recent call last):  File"<stdin>", line1, in<module>IOError:file is not ready for reading

>>>dbm.gnu.open("foox")Traceback (most recent call last):  File"<stdin>", line1, in<module>_gdbm.error:[Errno 2] No such file or directory

>>>imp.load_source("foo","foo",123)Traceback (most recent call last):  File"<stdin>", line1, in<module>IOError:[Errno 9] Bad file descriptor

>>>open("Python/","r")Traceback (most recent call last):  File"<stdin>", line1, in<module>IOError:[Errno 21] Is a directory: 'Python/'

>>>open("LICENSE").write("bar")Traceback (most recent call last):  File"<stdin>", line1, in<module>IOError:not writable>>>io.StringIO().fileno()Traceback (most recent call last):  File"<stdin>", line1, in<module>io.UnsupportedOperation:fileno>>>open("LICENSE").seek(1,1)Traceback (most recent call last):  File"<stdin>", line1, in<module>IOError:can't do nonzero cur-relative seeks

>>>f=open("LICENSE")>>>os.close(f.fileno())>>>f.read()Traceback (most recent call last):  File"<stdin>", line1, in<module>IOError:[Errno 9] Bad file descriptor

>>>f=open("LICENSE")>>>next(f)'A. HISTORY OF THE SOFTWARE\n'>>>f.tell()Traceback (most recent call last):  File"<stdin>", line1, in<module>IOError:telling position disabled by next() call

>>>mmap.mmap(123,10)Traceback (most recent call last):  File"<stdin>", line1, in<module>mmap.error:[Errno 9] Bad file descriptor>>>mmap.mmap(os.open("/tmp",os.O_RDONLY),10)Traceback (most recent call last):  File"<stdin>", line1, in<module>mmap.error:[Errno 13] Permission denied

>>>fd=os.open("LICENSE",os.O_RDONLY)>>>m=mmap.mmap(fd,16384)Traceback (most recent call last):  File"<stdin>", line1, in<module>WindowsError:[Error 5] Accès refusé>>>sys.last_value.errno13>>>errno.errorcode[13]'EACCES'>>>m=mmap.mmap(-1,4096)>>>m.resize(16384)Traceback (most recent call last):  File"<stdin>", line1, in<module>WindowsError:[Error 87] Paramètre incorrect>>>sys.last_value.errno22>>>errno.errorcode[22]'EINVAL'

>>>ossaudiodev.open("foo","r")Traceback (most recent call last):  File"<stdin>", line1, in<module>IOError:[Errno 2] No such file or directory: 'foo'

>>>readline.read_history_file("foo")Traceback (most recent call last):  File"<stdin>", line1, in<module>IOError:[Errno 2] No such file or directory>>>readline.read_init_file("foo")Traceback (most recent call last):  File"<stdin>", line1, in<module>IOError:[Errno 2] No such file or directory>>>readline.write_history_file("/dev/nonexistent")Traceback (most recent call last):  File"<stdin>", line1, in<module>IOError:[Errno 13] Permission denied

>>>signal.alarm(1);select.select([],[],[])0Traceback (most recent call last):  File"<stdin>", line1, in<module>select.error:(4, 'Interrupted system call')>>>e=sys.last_value>>>eerror(4, 'Interrupted system call')>>>e.errno==errno.EINTRTraceback (most recent call last):  File"<stdin>", line1, in<module>AttributeError:'error' object has no attribute 'errno'>>>e.args[0]==errno.EINTRTrue