Feature or enhancement

Proposal:

Introduction

Memory buffers used by native libraries often come with alignment requirements, such as page alignment. If an API dealing with such buffers is exposed into Python user code, the Python side is now responsible for providing a suitably aligned memory buffer. However, there is no way of properly specifying the alignment of any object supporting the buffer protocol, neither high-level ones likebytes andbytearray, nor lower-level ones likectypes arrays andmmap.

Over 10 years ago, aStackOverflow question has been asked for this exact problem, and one answer provides the most compact known workaround available to many Python versions. An additional problem it needs to consider here is that accessing the memory address of a Python buffer is nontrivial.

size=getBufferSizeFromSomewhere()alignment=getBufferAlignmentFromSomewhere()importctypes# Maximum amount of extra memory required - we can't know exactly before allocating!requiredOversize=size+alignment-1oversizeBuffer=bytearray(requiredOversize)# Get the address of the bytearray's backing bufferoversizeCType=ctypes.c_char*requiredOversizeoversizeMemory=oversizeCType.from_buffer(oversizeBuffer)oversizeAddress=ctypes.addressof(oversizeMemory)# Calculate required offset into oversized buffer to reach proper alignmentoffsetToAligned=alignment-oversizeAddress%alignment# Create a raw C buffer with offset to not copy the byte array in the processbufferCType=ctypes.c_char* (requiredOversize-offsetToAligned)correctRawCMemory=bufferCType.from_buffer(oversizeMemory,offsetToAligned)# Now use correctRawCMemory...

The C API gainedPyMem_AlignedAlloc() in 3.7 andnumpy also had a feature request accepted for aligned arrays.

Proposal

Note that I would like to find a better place for an aligned memory API, I just couldn't find one so far. See "Alternatives" below.

On all platforms supported by themmap module, add a new parameteralign to themmap.mmap function.

# Windowsmmap.mmap(fileno,length,tagname=None,access=ACCESS_DEFAULT,align=1[,offset])# Unixmmap.mmap(fileno,length,flags=MAP_SHARED,prot=PROT_WRITE|PROT_READ,access=ACCESS_DEFAULT,align=1[,offset])

If the mapping is a named file mapping (positive file descriptor number), the file's contents are mapped into memory starting from an address that is a multiple ofalign. If the mapping is an anonymous mapping (file descriptor -1), memory is allocated at an address that is a multiple ofalign. The default of 1 guarantees backwards compatibility, since it allows mapping to any address as before. Theoffset parameter only affects the start offset into the source data and has no effect on the starting address of the mapping.

Alternatives

Add memory alignment control to ctypes instead of mmap

The only place where I can see this would fit isctypes._CData.from_buffer_copy. Since ctypes does not provide a direct allocation API, the buffer needs to be allocated twice: once by the buffer provider (e.g. bytearray), and once for copying by ctypes. It also seems like a less convenient and obvious API than the other proposals

Add memory alignment control to bytearray

It is debatable whether adding such a low-level control feature to a high-level API like bytearray is a good design. However, this would make the feature available on Emscripten and WASI.

Has this already been discussed elsewhere?

This is a minor feature, which does not need previous discussion elsewhere

Links to previous discussion of this feature:

No response