Certain objects available in Python wrap access to an underlying memoryarray orbuffer. Such objects include the built-inbytes andbytearray, and some extension types likearray.array.Third-party libraries may define their own types for special purposes, suchas image processing or numeric analysis.
While each of these types have their own semantics, they share the commoncharacteristic of being backed by a possibly large memory buffer. It isthen desirable, in some situations, to access that buffer directly andwithout intermediate copying.
Python provides such a facility at the C level in the form of thebufferprotocol. This protocol has two sides:
Simple objects such asbytes andbytearray expose theirunderlying buffer in byte-oriented form. Other forms are possible; for example,the elements exposed by aarray.array can be multi-byte values.
An example consumer of the buffer interface is thewrite()method of file objects: any object that can export a series of bytes throughthe buffer interface can be written to a file. Whilewrite() onlyneeds read-only access to the internal contents of the object passed to it,other methods such asreadinto() need write accessto the contents of their argument. The buffer interface allows objects toselectively allow or reject exporting of read-write and read-only buffers.
There are two ways for a consumer of the buffer interface to acquire a bufferover a target object:
In both cases,PyBuffer_Release() must be called when the bufferisn’t needed anymore. Failure to do so could lead to various issues such asresource leaks.
Buffer structures (or simply “buffers”) are useful as a way to expose thebinary data from another object to the Python programmer. They can also beused as a zero-copy slicing mechanism. Using their ability to reference ablock of memory, it is possible to expose any data to the Python programmerquite easily. The memory could be a large, constant array in a C extension,it could be a raw block of memory for manipulation before passing to anoperating system library, or it could be used to pass around structured datain its native, in-memory format.
Contrary to most data types exposed by the Python interpreter, buffersare notPyObject pointers but rather simple C structures. Thisallows them to be created and copied very simply. When a generic wrapperaround a buffer is needed, amemoryview objectcan be created.
For short instructions how to write an exporting object, seeBuffer Object Structures. For obtaininga buffer, seePyObject_GetBuffer().
A pointer to the start of the logical structure described by the bufferfields. This can be any location within the underlying physical memoryblock of the exporter. For example, with negativestridesthe value may point to the end of the memory block.
For contiguous arrays, the value points to the beginning of the memoryblock.
A new reference to the exporting object. The reference is owned bythe consumer and automatically decremented and set toNULL byPyBuffer_Release(). The field is the equivalent of the returnvalue of any standard C-API function.
As a special case, fortemporary buffers that are wrapped byPyMemoryView_FromBuffer() orPyBuffer_FillInfo()this field isNULL. In general, exporting objects MUST NOTuse this scheme.
product(shape)*itemsize. For contiguous arrays, this is the lengthof the underlying memory block. For non-contiguous arrays, it is the lengththat the logical structure would have if it were copied to a contiguousrepresentation.
Accessing((char*)buf)[0]upto((char*)buf)[len-1] is only validif the buffer has been obtained by a request that guarantees contiguity. Inmost cases such a request will bePyBUF_SIMPLE orPyBUF_WRITABLE.
An indicator of whether the buffer is read-only. This field is controlledby thePyBUF_WRITABLE flag.
Item size in bytes of a single element. Same as the value ofstruct.calcsize()called on non-NULLformat values.
Important exception: If a consumer requests a buffer without thePyBUF_FORMAT flag,format willbe set toNULL, butitemsize still hasthe value for the original format.
Ifshape is present, the equalityproduct(shape)*itemsize==len still holds and the consumercan useitemsize to navigate the buffer.
Ifshape isNULL as a result of aPyBUF_SIMPLEor aPyBUF_WRITABLE request, the consumer must disregarditemsize and assumeitemsize==1.
ANUL terminated string instruct module style syntax describingthe contents of a single item. If this isNULL,"B" (unsigned bytes)is assumed.
This field is controlled by thePyBUF_FORMAT flag.
The number of dimensions the memory represents as an n-dimensional array.If it is 0,buf points to a single item representinga scalar. In this case,shape,stridesandsuboffsets MUST beNULL.
The macroPyBUF_MAX_NDIM limits the maximum number of dimensionsto 64. Exporters MUST respect this limit, consumers of multi-dimensionalbuffers SHOULD be able to handle up toPyBUF_MAX_NDIM dimensions.
An array ofPy_ssize_t of lengthndimindicating the shape of the memory as an n-dimensional array. Note thatshape[0]*...*shape[ndim-1]*itemsize MUST be equal tolen.
Shape values are restricted toshape[n]>=0. The caseshape[n]==0 requires special attention. Seecomplex arraysfor further information.
The shape array is read-only for the consumer.
An array ofPy_ssize_t of lengthndimgiving the number of bytes to skip to get to a new element in eachdimension.
Stride values can be any integer. For regular arrays, strides areusually positive, but a consumer MUST be able to handle the casestrides[n]<=0. Seecomplex arrays for further information.
The strides array is read-only for the consumer.
An array ofPy_ssize_t of lengthndim.Ifsuboffsets[n]>=0, the values stored along the nth dimension arepointers and the suboffset value dictates how many bytes to add to eachpointer after de-referencing. A suboffset value that is negativeindicates that no de-referencing should occur (striding in a contiguousmemory block).
This type of array representation is used by the Python Imaging Library(PIL). Seecomplex arrays for further information how to access elementsof such an array.
The suboffsets array is read-only for the consumer.
This is for use internally by the exporting object. For example, thismight be re-cast as an integer by the exporter and used to store flagsabout whether or not the shape, strides, and suboffsets arrays must befreed when the buffer is released. The consumer MUST NOT alter thisvalue.
Buffers are usually obtained by sending a buffer request to an exportingobject viaPyObject_GetBuffer(). Since the complexity of the logicalstructure of the memory can vary drastically, the consumer uses theflagsargument to specify the exact buffer type it can handle.
AllPy_buffer fields are unambiguously defined by the requesttype.
The following fields are not influenced byflags and must always be filled inwith the correct values:obj,buf,len,itemsize,ndim.
PyBUF_WRITABLE can be |’d to any of the flags in the next section.SincePyBUF_SIMPLE is defined as 0,PyBUF_WRITABLEcan be used as a stand-alone flag to request a simple writable buffer.
PyBUF_FORMAT can be |’d to any of the flags exceptPyBUF_SIMPLE.The latter already implies formatB (unsigned bytes).
The flags that control the logical structure of the memory are listedin decreasing order of complexity. Note that each flag contains all bitsof the flags below it.
| Request | shape | strides | suboffsets |
|---|---|---|---|
| yes | yes | if needed |
| yes | yes | NULL |
| yes | NULL | NULL |
| NULL | NULL | NULL |
C or Fortran contiguity can be explicitly requested, with and without strideinformation. Without stride information, the buffer must be C-contiguous.
| Request | shape | strides | suboffsets | contig |
|---|---|---|---|---|
| yes | yes | NULL | C |
| yes | yes | NULL | F |
| yes | yes | NULL | C or F |
| yes | NULL | NULL | C |
All possible requests are fully defined by some combination of the flags inthe previous section. For convenience, the buffer protocol provides frequentlyused combinations as single flags.
In the following tableU stands for undefined contiguity. The consumer wouldhave to callPyBuffer_IsContiguous() to determine contiguity.
| Request | shape | strides | suboffsets | contig | readonly | format |
|---|---|---|---|---|---|---|
| yes | yes | if needed | U | 0 | yes |
| yes | yes | if needed | U | 1 or 0 | yes |
| yes | yes | NULL | U | 0 | yes |
| yes | yes | NULL | U | 1 or 0 | yes |
| yes | yes | NULL | U | 0 | NULL |
| yes | yes | NULL | U | 1 or 0 | NULL |
| yes | NULL | NULL | C | 0 | NULL |
| yes | NULL | NULL | C | 1 or 0 | NULL |
The logical structure of NumPy-style arrays is defined byitemsize,ndim,shape andstrides.
Ifndim==0, the memory location pointed to bybuf isinterpreted as a scalar of sizeitemsize. In that case,bothshape andstrides areNULL.
Ifstrides isNULL, the array is interpreted asa standard n-dimensional C-array. Otherwise, the consumer must access ann-dimensional array as follows:
ptr=(char*)buf+indices[0]*strides[0]+...+indices[n-1]*strides[n-1]item=*((typeof(item)*)ptr);
As noted above,buf can point to any location withinthe actual memory block. An exporter can check the validity of a buffer withthis function:
defverify_structure(memlen,itemsize,ndim,shape,strides,offset):"""Verify that the parameters represent a valid array within the bounds of the allocated memory: char *mem: start of the physical memory block memlen: length of the physical memory block offset: (char *)buf - mem """ifoffset%itemsize:returnFalseifoffset<0oroffset+itemsize>memlen:returnFalseifany(v%itemsizeforvinstrides):returnFalseifndim<=0:returnndim==0andnotshapeandnotstridesif0inshape:returnTrueimin=sum(strides[j]*(shape[j]-1)forjinrange(ndim)ifstrides[j]<=0)imax=sum(strides[j]*(shape[j]-1)forjinrange(ndim)ifstrides[j]>0)return0<=offset+iminandoffset+imax+itemsize<=memlen
In addition to the regular items, PIL-style arrays can contain pointersthat must be followed in order to get to the next element in a dimension.For example, the regular three-dimensional C-arraycharv[2][2][3] canalso be viewed as an array of 2 pointers to 2 two-dimensional arrays:char(*v[2])[2][3]. In suboffsets representation, those two pointerscan be embedded at the start ofbuf, pointingto twocharx[2][3] arrays that can be located anywhere in memory.
Here is a function that returns a pointer to the element in an N-D arraypointed to by an N-dimensional index when there are both non-NULL stridesand suboffsets:
void*get_item_pointer(intndim,void*buf,Py_ssize_t*strides,Py_ssize_t*suboffsets,Py_ssize_t*indices){char*pointer=(char*)buf;inti;for(i=0;i<ndim;i++){pointer+=strides[i]*indices[i];if(suboffsets[i]>=0){pointer=*((char**)pointer)+suboffsets[i];}}return(void*)pointer;}
Return 1 ifobj supports the buffer interface otherwise 0. When 1 isreturned, it doesn’t guarantee thatPyObject_GetBuffer() willsucceed.
Send a request toexporter to fill inview as specified byflags.If the exporter cannot provide a buffer of the exact type, it MUST raisePyExc_BufferError, setview->obj toNULL andreturn -1.
On success, fill inview, setview->obj to a new referencetoexporter and return 0. In the case of chained buffer providersthat redirect requests to a single object,view->obj MAYrefer to this object instead ofexporter (SeeBuffer Object Structures).
Successful calls toPyObject_GetBuffer() must be paired with callstoPyBuffer_Release(), similar tomalloc() andfree().Thus, after the consumer is done with the buffer,PyBuffer_Release()must be called exactly once.
Release the bufferview and decrement the reference count forview->obj. This function MUST be called when the bufferis no longer being used, otherwise reference leaks may occur.
It is an error to call this function on a buffer that was not obtained viaPyObject_GetBuffer().
Return the implieditemsize fromformat.This function is not yet implemented.
Return 1 if the memory defined by theview is C-style (order is'C') or Fortran-style (order is'F') contiguous or either one(order is'A'). Return 0 otherwise.
Fill thestrides array with byte-strides of a contiguous (C-style iforder is'C' or Fortran-style iforder is'F') array of thegiven shape with the given number of bytes per element.
Handle buffer requests for an exporter that wants to exposebuf of sizelenwith writability set according toreadonly.buf is interpreted as a sequenceof unsigned bytes.
Theflags argument indicates the request type. This function always fills inview as specified by flags, unlessbuf has been designated as read-onlyandPyBUF_WRITABLE is set inflags.
On success, setview->obj to a new reference toexporter andreturn 0. Otherwise, raisePyExc_BufferError, setview->obj toNULL and return -1;
If this function is used as part of agetbufferproc,exporter MUST be set to the exporting object. Otherwise,exporter MUSTbe NULL.
Enter search terms or a module, class or function name.
