Abstract

This PEP proposes an extension to the buffer interface called the‘locked buffer interface’.

The locked buffer interface avoids the flaws of the ‘old’ bufferinterface[1] as defined in Python versions up to and including2.2, and has the following semantics:

• The lifetime of the retrieved pointer is clearly defined andcontrolled by the client.
• The buffer size is returned as a ‘size_t’ data type, whichallows access to large buffers on platforms wheresizeof(int)!=sizeof(void*).

(Guido comments: This second sounds like a change we could alsomake to the “old” buffer interface, if we introduce another flagbit that’snot part of the default flags.)

Specification

The locked buffer interface exposes new functions which return thesize and the pointer to the internal memory block of any pythonobject which chooses to implement this interface.

Retrieving a buffer from an object puts this object in a lockedstate during which the buffer may not be freed, resized, orreallocated.

The object must be unlocked again by releasing the buffer if it’sno longer used by calling another function in the locked bufferinterface. If the object never resizes or reallocates the bufferduring its lifetime, this function may be NULL. Failure to callthis function (if it is != NULL) is a programming error and mayhave unexpected results.

The locked buffer interface omits the memory segment model whichis present in the old buffer interface - only a single memoryblock can be exposed.

The memory blocks can be accessed without holding the globalinterpreter lock.

Implementation

Define a new flag in Include/object.h:

/*PyBufferProcscontainsbf_acquirelockedreadbuffer,bf_acquirelockedwritebuffer,andbf_releaselockedbuffer*/#define Py_TPFLAGS_HAVE_LOCKEDBUFFER (1L<<15)

This flag would be included inPy_TPFLAGS_DEFAULT:

#define Py_TPFLAGS_DEFAULT  ( \....Py_TPFLAGS_HAVE_LOCKEDBUFFER| \....0)

Extend thePyBufferProcs structure by new fields inInclude/object.h:

typedefsize_t(*acquirelockedreadbufferproc)(PyObject*,constvoid**);typedefsize_t(*acquirelockedwritebufferproc)(PyObject*,void**);typedefvoid(*releaselockedbufferproc)(PyObject*);typedefstruct{getreadbufferprocbf_getreadbuffer;getwritebufferprocbf_getwritebuffer;getsegcountprocbf_getsegcount;getcharbufferprocbf_getcharbuffer;/*lockedbufferinterfacefunctions*/acquirelockedreadbufferprocbf_acquirelockedreadbuffer;acquirelockedwritebufferprocbf_acquirelockedwritebuffer;releaselockedbufferprocbf_releaselockedbuffer;}PyBufferProcs;

The new fields are present if thePy_TPFLAGS_HAVE_LOCKEDBUFFERflag is set in the object’s type.

ThePy_TPFLAGS_HAVE_LOCKEDBUFFER flag implies thePy_TPFLAGS_HAVE_GETCHARBUFFER flag.

Theacquirelockedreadbufferproc andacquirelockedwritebufferprocfunctions return the size in bytes of the memory block on success,and fill in the passed void * pointer on success. If thesefunctions fail - either because an error occurs or no memory blockis exposed - they must set the void * pointer to NULL and raise anexception. The return value is undefined in these cases andshould not be used.

If calls to these functions succeed, eventually the buffer must bereleased by a call to thereleaselockedbufferproc, supplying theoriginal object as argument. Thereleaselockedbufferproc cannotfail. For objects that actually maintain an internal lock countit would be a fatal error if thereleaselockedbufferproc functionwould be called too often, leading to a negative lock count.

Similar to the ‘old’ buffer interface, any of these functions maybe set to NULL, but it is strongly recommended to implement thereleaselockedbufferproc function (even if it does nothing) if anyof theacquireread/writelockedbufferproc functions areimplemented, to discourage extension writers from checking for aNULL value and not calling it.

These functions aren’t supposed to be called directly, they arecalled through convenience functions declared inInclude/abstract.h:

intPyObject_AcquireLockedReadBuffer(PyObject*obj,constvoid**buffer,size_t*buffer_len);intPyObject_AcquireLockedWriteBuffer(PyObject*obj,void**buffer,size_t*buffer_len);voidPyObject_ReleaseLockedBuffer(PyObject*obj);

The former two functions return 0 on success, set buffer to thememory location and buffer_len to the length of the memory blockin bytes. On failure, or if the locked buffer interface is notimplemented by obj, they return -1 and set an exception.

The latter function doesn’t return anything, and cannot fail.

Backward Compatibility

The size of thePyBufferProcs structure changes if this proposalis implemented, but the type’stp_flags slot can be used todetermine if the additional fields are present.

Reference Implementation

An implementation has been uploaded to the SourceForge patchmanager ashttps://bugs.python.org/issue652857.

Additional Notes/Comments

Python strings, unicode strings, mmap objects, and array objectswould expose the locked buffer interface.

mmap and array objects would actually enter a locked state whilethe buffer is active, this is not needed for strings and unicodeobjects. Resizing locked array objects is not allowed and willraise an exception. Whether closing a locked mmap object is anerror or will only be deferred until the lock count reaches zerois an implementation detail.

Guido recommends

But I’m still very concerned that if most built-in types(e.g. strings, bytes) don’t implement the releasefunctionality, it’s too easy for an extension to seem to workwhile forgetting to release the buffer.

I recommend that at least some built-in types implement theacquire/release functionality with a counter, and assert thatthe counter is zero when the object is deleted – if theassert fails, someone DECREF’ed their reference to the objectwithout releasing it. (The rule should be that you must own areference to the object while you’ve acquired the object.)

For strings that might be impractical because the stringobject would have to grow 4 bytes to hold the counter; but thenew bytes object (PEP 296) could easily implement the counter,and the array object too – that way there will be plenty ofopportunity to test proper use of the protocol.

Community Feedback

Greg Ewing doubts the locked buffer interface is needed at all, hethinks the normal buffer interface could be used if the pointer is(re)fetched each time it’s used. This seems to be dangerous,because even innocent looking calls to the Python API likePy_DECREF() may trigger execution of arbitrary Python code.

The first version of this proposal didn’t have the releasefunction, but it turned out that this would have been toorestrictive: mmap and array objects wouldn’t have been able toimplement it, because mmap objects can be closed anytime if notlocked, and array objects could resize or reallocate the buffer.

This PEP will probably be rejected because nobody except theauthor needs it.

References

Copyright

This document has been placed in the public domain.