Important
This PEP is a historical document. The up-to-date, canonical documentation can now be found at thePyBytesWriter API.
×
SeePEP 1 for how to propose changes.
Add a newPyBytesWriter C API to createbytes objects.
Soft deprecatePyBytes_FromStringAndSize(NULL,size) and_PyBytes_Resize() APIs. These APIs treat an immutablebytesobject as a mutable object. They remain available and maintained, don’temit deprecation warning, but are no longer recommended when writing newcode.
Creating a Pythonbytes object usingPyBytes_FromStringAndSize(NULL,size) and_PyBytes_Resize()treats an immutablebytes object as mutable. It goes againstthe principle thatbytes objects are immutable. It also createsan incomplete or “invalid” object since bytes are not initialized. InPython, abytes object should always have its bytes fullyinitialized.
When creating a bytes string and the output size is unknown, onestrategy is to allocate a short buffer and extend it (to the exact size)each time a larger write is needed.
This strategy is inefficient because it requires enlarging the buffermultiple times. It’s more efficient to overallocate the buffer thefirst time that a larger write is needed. It reduces the number ofexpensiverealloc() operations which can imply a memory copy.
bytes writer instance created byPyBytesWriter_Create().The instance must be destroyed byPyBytesWriter_Finish() orPyBytesWriter_Discard().
PyBytesWriter to writesize bytes.Ifsize is greater than zero, allocatesize bytes, and set thewriter size tosize. The caller is responsible to writesizebytes usingPyBytesWriter_GetData().
On error, set an exception and return NULL.
size must be positive or zero.
PyBytesWriter created byPyBytesWriter_Create().On success, return a Pythonbytes object.On error, set an exception and returnNULL.
The writer instance is invalid after the call in any case.
PyBytesWriter_Finish(), but resize the writertosize bytes before creating thebytes object.PyBytesWriter_Finish(), but resize the writerusingbuf pointer before creating thebytes object.Set an exception and returnNULL ifbuf pointer is outside theinternal buffer bounds.
Function pseudo-code:
Py_ssize_tsize=(char*)buf-(char*)PyBytesWriter_GetData(writer);returnPyBytesWriter_FinishWithSize(writer,size);
PyBytesWriter created byPyBytesWriter_Create().Do nothing ifwriter isNULL.
The writer instance is invalid after the call.
Ifsize is equal to-1, callstrlen(bytes) to get thestring length.
On success, return0.On error, set an exception and return-1.
PyBytes_FromFormat(), but write the output directly atthe writer end. Grow the writer internal buffer on demand.Then add the written size to the writer size.On success, return0.On error, set an exception and return-1.
The pointer is valid untilPyBytesWriter_Finish() orPyBytesWriter_Discard() is called onwriter.
Newly allocated bytes are left uninitialized.
On success, return0.On error, set an exception and return-1.
size must be positive or zero.
Newly allocated bytes are left uninitialized.
On success, return0.On error, set an exception and return-1.
size can be negative to shrink the writer.
PyBytesWriter_Grow(), but update also thebufpointer.Thebuf pointer is moved if the internal buffer is moved in memory.Thebuf relative position within the internal buffer is leftunchanged.
On error, set an exception and returnNULL.
buf must not beNULL.
Function pseudo-code:
Py_ssize_tpos=(char*)buf-(char*)PyBytesWriter_GetData(writer);if(PyBytesWriter_Grow(writer,size)<0){returnNULL;}return(char*)PyBytesWriter_GetData(writer)+pos;
PyBytesWriter_Resize() andPyBytesWriter_Grow()overallocate the internal buffer to reduce the number ofrealloc()calls and so reduce memory copies.
PyBytesWriter_Finish() trims overallocations: it shrinks theinternal buffer to the exact size when creating the finalbytesobject.
The API is not thread safe: a writer should only be used by a singlethread at the same time.
Soft deprecatePyBytes_FromStringAndSize(NULL,size) and_PyBytes_Resize() APIs. These APIs treat an immutablebytesobject as a mutable object. They remain available and maintained, don’temit deprecation warning, but are no longer recommended when writing newcode.
PyBytes_FromStringAndSize(str,size) is not soft deprecated. Onlycalls withNULLstr are soft deprecated.
Create the bytes stringb"HelloWorld!":
PyObject*hello_world(void){PyBytesWriter*writer=PyBytesWriter_Create(0);if(writer==NULL){gotoerror;}if(PyBytesWriter_WriteBytes(writer,"Hello",-1)<0){gotoerror;}if(PyBytesWriter_Format(writer," %s!","World")<0){gotoerror;}returnPyBytesWriter_Finish(writer);error:PyBytesWriter_Discard(writer);returnNULL;}
Example creating the bytes stringb"abc", with a fixed size of 3 bytes:
PyObject*create_abc(void){PyBytesWriter*writer=PyBytesWriter_Create(3);if(writer==NULL){returnNULL;}char*str=PyBytesWriter_GetData(writer);memcpy(str,"abc",3);returnPyBytesWriter_Finish(writer);}
GrowAndUpdatePointer() exampleExample using a pointer to write bytes and to track the written size.
Create the bytes stringb"HelloWorld":
PyObject*grow_example(void){// Allocate 10 bytesPyBytesWriter*writer=PyBytesWriter_Create(10);if(writer==NULL){returnNULL;}// Write some byteschar*buf=PyBytesWriter_GetData(writer);memcpy(buf,"Hello ",strlen("Hello "));buf+=strlen("Hello ");// Allocate 10 more bytesbuf=PyBytesWriter_GrowAndUpdatePointer(writer,10,buf);if(buf==NULL){PyBytesWriter_Discard(writer);returnNULL;}// Write more bytesmemcpy(buf,"World",strlen("World"));buf+=strlen("World");// Truncate the string at 'buf' position// and create a bytes objectreturnPyBytesWriter_FinishWithPointer(writer,buf);}
PyBytes_FromStringAndSize() codeExample of code using the soft deprecatedPyBytes_FromStringAndSize(NULL,size) API:
PyObject*result=PyBytes_FromStringAndSize(NULL,num_bytes);if(result==NULL){returnNULL;}if(copy_bytes(PyBytes_AS_STRING(result),start,num_bytes)<0){Py_CLEAR(result);}returnresult;
It can now be updated to:
PyBytesWriter*writer=PyBytesWriter_Create(num_bytes);if(writer==NULL){returnNULL;}if(copy_bytes(PyBytesWriter_GetData(writer),start,num_bytes)<0){PyBytesWriter_Discard(writer);returnNULL;}returnPyBytesWriter_Finish(writer);
_PyBytes_Resize() codeExample of code using the soft deprecated_PyBytes_Resize() API:
PyObject*v=PyBytes_FromStringAndSize(NULL,size);if(v==NULL){returnNULL;}char*p=PyBytes_AS_STRING(v);// ... fill bytes into 'p' ...if(_PyBytes_Resize(&v,(p-PyBytes_AS_STRING(v)))){returnNULL;}returnv;
It can now be updated to:
PyBytesWriter*writer=PyBytesWriter_Create(size);if(writer==NULL){returnNULL;}char*p=PyBytesWriter_GetData(writer);// ... fill bytes into 'p' ...returnPyBytesWriter_FinishWithPointer(writer,p);
Notes on the CPython reference implementation which are not part of theSpecification:
bytes object, soPyBytesWriter_Finish() just returns the object without havingto copy memory.bytes object which isinefficient. At the end,PyBytesWriter_Finish() creates thebytes object from this small buffer.PyBytesWriter on the heap memory.There is no impact on the backward compatibility, only new APIs areadded.
PyBytes_FromStringAndSize(NULL,size) and_PyBytes_Resize() APIsare soft deprecated. No new warnings is emitted when these functions areused and they are not planned for removal.
_PyBytesWriter C API.This document is placed in the public domain or under theCC0-1.0-Universal license, whichever is more permissive.
Source:https://github.com/python/peps/blob/main/peps/pep-0782.rst
Last modified:2025-09-18 13:28:58 GMT