PEP 688, take 2: Making the buffer protocol accessible in Python

PEPs
1

I just posted a revised version ofPEP 688, which now proposes a Python-level API for the buffer protocol, adding the new__buffer__ and__release_buffer__ dunders. This allows for buffer protocol support in the type system, and also allows Python classes to implement the buffer protocol.

The previous discussion was here:PEP 688: Making the buffer protocol accessible in Python

Possible topics of discussion:

  • The PEP still proposes removing the special case where “bytes” is supposed to also mean “bytearray” and “memoryview” in annotations. This proposal got some pushback in the previous thread.
  • Does the interaction between the C buffer API and the proposed Python API make sense?
  • A few opportunities for bikeshedding: where should theBuffer ABC live, what should the new special methods be called, should we add any other related useful tools to the standard library?
4 Likes
2

As this feature is a feature of the language, as well as of the C-API, is it possible to specifythe Python behaviour without reference to the implementation language, then the change to the C-API and slots separately? I’m not arguing against discussing the reference implementation as evidence of feasibility.

My interest in this is not primarily typing (the category the PEP has).

The example is not enough for my understanding of the possibilities in general because it is a special case. Taking thememoryview in the constructor shape-locks thebytearray for the lifetime of the object by upping the export count, but suppose we wanted not to?

LetMyBuffer admit anappend operation delegated to thebytearray, presumably only possible when theMyBuffer is not exporting. The__buffer__ method would have to return a freshmemoryview, and as long as any had not been returned to__release_buffer__, callingappend would fail, right?

MyBuffer.__release_buffer__ would have to callmemoryview.release. I think I understand this is not the samememoryview handed out, although it wraps the samePy_buffer.

It seems like, in this story, theMyBuffer is not conscious of its export count, since thebytearray takes care of it invisibly. It could track them if it wanted to by pairing off the calls, but not in a boolean as now.

Would this make a good mutable example?

3

Thanks for the feedback! I put up a PR atPEP 688: Enhancements from discussion by JelleZijlstra · Pull Request #2832 · python/peps · GitHub that

  • extends the specification to focus more on the behavior, not on the C slots
  • expands theMyBuffer example with anextend method similar to what you outline

Let me know if this addresses your concerns.

4

Yes, I think that’s better for starting with the Python behaviour.

It is challenging to follow what exactly is happening with wrapping in amemoryview. However, these specifications always need careful reading, and one’s interpretation to be checked with the reference implementation.

What I expect to see in a hypothetical referenceslot_bf_buffer, for example, is code that begins likevectorcall_method(&_Py_ID(__buffer__), ...) and discards thememoryview wrapper it gets back. And inslot_bf_buffer_release I expect code that synthesises a newmemoryview wrapper around the passedPy_buffer.

This doesn’t seem quite what is implied by:

When this method is invoked through the buffer API (for example, throughmemoryview.release), the passedmemoryview is the same object as was returned by__buffer__.

The signatures ofslot_bf_buffer andslot_bf_buffer_release are:

typedef int (*getbufferproc)(PyObject *, Py_buffer *, int);typedef void (*releasebufferproc)(PyObject *, Py_buffer *);

so there is no opportunity to guarantee as you have inMyBuffer.__realease_buffer__ that:

    def __release_buffer__(self, view: memoryview) -> None:        assert self.view is view  # guaranteed to be true        ...

If I use thePython API, calling__buffer__and__release_buffer__ myself, then I have to pass exactly thememoryview I received, since I have no other way to be sure I an returning the samePy_buffer. But whenmemoryview.release does this to a buffer object defined in Python, it will call the slot containingslot_bf_buffer_release and the buffer object in Python will receive the new ephemeralmemoryview.

5

Sorry for bad news, but:

if a buffer is writable, it must define thebf_releasebuffer slot, so that the buffer can be released when a consumer is done writing to it.

That’s not actually true. Thebf_releasebuffer slot is needed when there’s extra cleanup to be done, other than decref(view->obj). That’s orthogonal to the data being writable (PyBUF_WRITABLE).
For example, NumPy arrays are mutable, butsetbf_releasebuffer to NULL.
Types with mutable“shape”, likebytearray which hasappend, generallyneed to implementbf_releasebuffer so they can prevent resizing while a buffer is exported. But I don’t think “mutable shape” is a very useful property for this PEP, and anyway it’s not the only possible reason for implementingbf_releasebuffer.

I believe this came from reading theargument parsing docs? The wording there is unfortunate – by a stretch, one could argue it’s technically correct (if you read it as giving a specific local definition to the terms “mutable”/“read-only”), but it definitely suggests wrong stuff for uses outside arg parsing.

I guesstp_as_buffer could grow field(s) for “not supported” and/or “~guaranteed support”PyBUF* flags? At the moment it’s really only available at runtime byEAFP, as far as I know. Or perhaps it’s not useful for the C API, and could just be Python class attributes or some kind of typing-only hints.

6

Thanks for bringing this up! I got this idea from a post in the previous thread:PEP 688: Making the buffer protocol accessible in Python - #29 by storchaka. It took me a while to convince myself that it’s true, but maybe it’s not! The line of thinking that persuaded me was that anything that allows a writable buffer must keep track of whether someone is already writing to it in order to be thread-safe, and therefore it must have abf_releasebuffer slot. Among the stdlib’s buffer classes, it also seems to be true that all the mutable ones havebf_releasebuffer slots. Numpy gets around this with manual refcounting: the buffer returned bybf_getbuffer holds a reference that the consumer must eventually DECREF, which signals to the numpy array that its consumer is gone.

If we can’t usebf_releasebuffer to signal mutability, I think we’ll have to go back to a singleBuffer type, with no affordance for mutability in the type system. There just isn’t an elegant way to support it.

7

The current implementation does indeed provide this guarantee, and that’s why I am comfortable making this guarantee in the spec. The prototype is here:cpython/typeobject.c at de3a4bc518f7137a7a3c86cc7fc28b70fcb42013 · JelleZijlstra/cpython · GitHub

I implemented it this way because we need to ensure that when the consumer of the buffer created through__buffer__ releases it, we call__release_buffer__ on the same Python object. But if we just returned the buffer inside the memoryview we got from__buffer__, there’s no way to get back to the original object. So the implementation creates another wrapper that holds a reference to the memoryview and the Python object. When the buffer is released, we properly call__release_buffer__ while also cleaning up whatever is inside the memoryview.

8

One use-case where a bf_releasebuffer for a read-only buffer might come up is accessing buffers managed by some other system, for example when interfacing with a system that uses a moving GC.

9
Jelle Zijlstra:

The current implementation does indeed provide this guarantee, and that’s why I am comfortable making this guarantee in the spec. …

I implemented it this way because we need to ensure that when the consumer of the buffer created through__buffer__ releases it, we call__release_buffer__ on the same Python object. But if we just returned the buffer inside the memoryview we got from__buffer__, there’s no way to get back to the original object.

Not the.obj of thePy_buffer being released? I know this can beNULL, in principle, but that’s when there isn’t meaningfully an underlying object. We are interested in the case of a C client, so it would callPyBuffer_Release(). That seems to manage ok given just aPy_buffer.

The guarantee seems to make this quite complicated, and I don’t see why it is needed.

Edit: Ok, I see the leap I’m making. The.buf of the buffer in thememoryview is thebytearray (in the example of use) not theMyBuffer instance itself. So you want a second layer to this whereview->obj inPyBuffer_Release(Py_buffer *view) is the theMyBuffer instance.

Edit 2: Having got that clear, I think it is now difficult to see how an object would give you, through the__buffer__ method, amemoryview that truly represented a buffer view of itself, rather than only ever one bytes-like member of itself, of a built-in type that was already capable of of the buffer interface. Is it useful enough if it only does that?

(Jeff Allen)10

I wonder if a good examples to illustrate the idea would be:

  1. objects that hold images in a compressed form, but will export their image as an array of decoded integer values.
  2. an audio equivalent of that.
  3. a sparse array that unpacks to an array whilst exported.

I looked atmmap as a possible example, and it expects to be released, whether read-only or not.

2 Likes
11
Jelle Zijlstra:

anything that allows a writable buffer must keep track of whether someone is already writing to it in order to be thread-safe

There’s no rule that objects must be thread-safe. Thread safety is a very nice property to have, sure, and I’d expect well-behaved objects to have it, but I’d also expectsome objects that expose raw memory to eschew it in favor of performance.

Also, while "one writeror N readers” is a good way to ensure safety, it isn’t strictly necessary. (Think of a buffer of results, where each result can be set independently by a different thread. It might not be too performant, but it should work.)

Jelle Zijlstra:

Among the stdlib’s buffer classes, it also seems to be true that all the mutable ones havebf_releasebuffer slots.

Yes. That definitely contributes to the confusion.

Jelle Zijlstra:

Numpy gets around this with manual refcounting: the buffer returned bybf_getbuffer holds a reference that the consumer must eventually DECREF, which signals to the numpy array that its consumer is gone.

All Python buffers hold that reference (view->obj).PyBuffer_ReleaseDECREFs it. There’s nothing manual about it from NumPy’s point of view.

Going back toC argument parsing which I think is the source of this confusion:

  • PyArg_ParseTuple can take an object withbf_releasebuffer=NULL and return its buffer asconst char*, because ofPyArg_ParseTuple’s very special semantics: its inputs outlive its outputs, and so it doesn’t need an INCREF (i.e. the returnedconst char* has a hidden borrowed reference).
  • PyArg_ParseTuple cannot take objects withbf_releasebuffer and convert them toconst char*, because it has no way to callbf_releasebuffer when the buffer is no longer needed. To handle these types, the user must ask forPy_buffer (or a Python object).

So, simple C functions (ones that usePyArg_ParseTuple and deal withconst char* internally) don’t allow types withbf_releasebuffer set, which means that they disallow common mutable types such asbytearray. It’s easy to read that as “mutable types havebf_releasebuffer set”, especially when there are no good counterexamples in the stdlib.

12

I decided to give up on distinguishing mutable vs. immutable buffers for now, especially becauseIntrospection and "mutable XOR shared" semantics for PyBuffer may give us a more robust way to tell them apart in the future. This will make the PEP simpler but still useful for typing. A new version of the PEP just live atPEP 688 – Making the buffer protocol accessible in Python | peps.python.org.

13

FYI, I filedgh-98712 to clarify the C arg parsing docs.

14

Unless some additional feedback comes up, I plan to submit the PEP to the SC in the next few weeks.

Over the last week I have been reviewingbytes type annotations in typeshed (Track review of `bytes` types · Issue #9006 · python/typeshed · GitHub); so far I have covered nearly all of the stdlib. This uncovered many more places where the standard library acceptsbytes but notbytearray, which in my mind strengthens the case for dropping the implicit conversion between the two. I posted some tweaks to the PEP (PEP 688: Small tweaks by JelleZijlstra · Pull Request #2866 · python/peps · GitHub) to reflect that.

2 Likes
15

I haven’t reviewed__release_buffer__ yet. I’ll need to find a chunk of time to wrap my head around it again.
Currently it’s not clear to me when to call__release_buffer__ vs. when a wrapper or memoryview does it for you: e.g. the PEP says “It is also possible to call__release_buffer__ on a C class that implementsbf_releasebuffer” – whenshould you do it, and what happens if you should do it but don’t (and vice versa)?

I took the reference implementation and addedassert (self->ob_exports >= 0); toarray_buffer_relbuf. That asserts a C-level invariant – it’s definitely not something I should be able to trigger through Python code. But I can:

>>> import array>>> a = array.array('b', range(5))>>> m = a.__buffer__(0)>>> a.__release_buffer__(m)>>> a.__release_buffer__(m)python: ./Modules/arraymodule.c:2606: array_buffer_relbuf: Assertion `self->ob_exports >= 0' failed.
1 Like
16

FWIW, when I get time I want to check if the following would be a better (simpler/safer) approach:

  • memoryview itself gets an extra field for “Python re-exporter”
  • A Python implementation of__buffer__ must returnmemoryview(..., exporter=self).
    • The__buffer__→tp_getbuffer wrapper checks this
  • memorview calls the re-exporter’s__release_buffer__ onrelease() (unless released already).
  • Thetp_releasebuffer→__release_buffer__ wrapper only checks the argument and callsrelease() on it, thus only calling code that’s safe to call from Python.
  • You never need to call__release_buffer__ manually, but if you do, nothing bad happens (there’ll probably be recursion/reentrancy issues to solve in guaranteeing this)
1 Like
17
Petr Viktorin:

Currently it’s not clear to me when to call__release_buffer__ vs. when a wrapper or memoryview does it for you: e.g. the PEP says “It is also possible to call__release_buffer__ on a C class that implementsbf_releasebuffer” – whenshould you do it, and what happens if you should do it but don’t (and vice versa)?

You should call it when you want to release a memoryview you got from a call to__buffer__. It’s probably going to be very rarely necessary, because you can just dowith obj.__buffer__(flags) and the memoryview’s__exit__ method will call the buffer for you. Similarly, if you don’t ever call it, the runtime will release the buffer for you when the memoryview you got from__buffer__ gets GCed. I will add some discussion of this to the PEP.

Petr Viktorin:

I took the reference implementation and addedassert (self->ob_exports >= 0); toarray_buffer_relbuf. That asserts a C-level invariant – it’s definitely not something I should be able to trigger through Python code. But I can:

Good catch! I’m testing a change to the reference implementation now that basically callsmemoryview.release fromwrap_releasebuffer, so that memoryview takes care of checking that we’re not re-releasing an already released buffer.

Petr Viktorin:

FWIW, when I get time I want to check if the following would be a better (simpler/safer) approach:

This isn’t too different from how the reference implementation already works. However, instead of setting a flag on the memoryview returned from__buffer__, I wrap it in another memoryview that remembers its Python exporter by setting itsobj to a wrapper type that references the Python exporter and the underlying memoryview that__buffer__ returned. This avoids changes to memoryview:

>>> class X:...     def __buffer__(self, flags): return memoryview(b"x")... >>> mv = memoryview(>>> mv = memoryview(X())>>> mv.obj<_buffer_wrapper object at 0x10120e670>>>> import gc>>> gc.get_referents(mv.obj)[<memory at 0x10114cd50>, <__main__.X object at 0x1011ab990>]
(Petr Viktorin)18
Jelle Zijlstra:

This avoids changes to memoryview

How important is that, really?

Jelle Zijlstra:

You should call it when you want to release a memoryview you got from a call to__buffer__. It’s probably going to be very rarely necessary, because you can just dowith obj.__buffer__(flags) and the memoryview’s__exit__ method will call the buffer for you.

It won’t:

>>> class Foo:...     def __buffer__(self, flags):...         print('__buffer__ called')...         return memoryview(b"here's your data")...     def __release_buffer__(self, view):...         print('important releasing code here')...         view.release()... >>> with Foo().__buffer__(0):...     pass... __buffer__ called>>> # release not called!

Hence the idea of requiringmemoryview(..., exporter=self) indef __buffer__.

(Jelle Zijlstra)19
Petr Viktorin:

How important is that, really?

I’d be open to adding new Python APIs if that’s helpful, but every new Python API has some cost (documentation, bikeshedding over naming/semantics, having to reason about correctness if it’s used in unexpected ways). So I’d prefer to avoid introducing a new Python API if possible.

Petr Viktorin:

It won’t:

>>> class Foo:...     def __buffer__(self, flags):...         print('__buffer__ called')...         return memoryview(b"here's your data")...     def __release_buffer__(self, view):...         print('important releasing code here')...         view.release()... >>> with Foo().__buffer__(0):...     pass... __buffer__ called>>> # release not called!

If you call__buffer__ directly, you’re on your own. I see this as similar to the context manager protocol:__enter__ and__exit__ should be called in pairs, but if you call__enter__ directly and don’t call__exit__, the runtime isn’t going to call it for you.

If you usewith memoryview(Foo()): instead,Foo.__release_buffer__ will be called.

Petr Viktorin:

Hence the idea of requiringmemoryview(..., exporter=self) indef __buffer__.

Such a requirement wouldn’t be enforceable in code that directly calls__buffer__ (at least, not without bigger changes to the language).