multiprocessing.shared_memory — Shared memory for direct access across processes¶

Source code:Lib/multiprocessing/shared_memory.py

This module provides a class,SharedMemory, for the allocationand management of shared memory to be accessed by one or more processeson a multicore or symmetric multiprocessor (SMP) machine. To assist withthe life-cycle management of shared memory especially across distinctprocesses, aBaseManager subclass,SharedMemoryManager, is also provided in themultiprocessing.managers module.

In this module, shared memory refers to “POSIX style” shared memory blocks(though is not necessarily implemented explicitly as such) and does not referto “distributed shared memory”. This style of shared memory permits distinctprocesses to potentially read and write to a common (or shared) region ofvolatile memory. Processes are conventionally limited to only have access totheir own process memory space but shared memory permits the sharingof data between processes, avoiding the need to instead send messages betweenprocesses containing that data. Sharing data directly via memory can providesignificant performance benefits compared to sharing data via disk or socketor other communications requiring the serialization/deserialization andcopying of data.

classmultiprocessing.shared_memory.SharedMemory(name=None,create=False,size=0)¶

Create an instance of theSharedMemory class for eithercreating a new shared memory block or attaching to an existing sharedmemory block. Each shared memory block is assigned a unique name.In this way, one process can create a shared memory block with aparticular name and a different process can attach to that same sharedmemory block using that same name.

As a resource for sharing data across processes, shared memory blocksmay outlive the original process that created them. When one processno longer needs access to a shared memory block that might still beneeded by other processes, theclose() method should be called.When a shared memory block is no longer needed by any process, theunlink() method should be called to ensure proper cleanup.

Parameters:

• name (str |None) – The unique name for the requested shared memory, specified as a string.When creating a new shared memory block, ifNone (the default)is supplied for the name, a novel name will be generated.

• create (bool) – Control whether a new shared memory block is created (True)or an existing shared memory block is attached (False).

• size (int) – The requested number of bytes when creating a new shared memory block.Because some platforms choose to allocate chunks of memorybased upon that platform’s memory page size, the exact size of the sharedmemory block may be larger or equal to the size requested.When attaching to an existing shared memory block,thesize parameter is ignored.

close()¶

Close access to the shared memory from this instance. In order toensure proper cleanup of resources, all instances should callclose() once the instance is no longer needed. Note that callingclose() does not cause the shared memory block itself to bedestroyed.

unlink()¶

Request that the underlying shared memory block be destroyed. Inorder to ensure proper cleanup of resources,unlink() should becalled once (and only once) across all processes which have needfor the shared memory block. After requesting its destruction, ashared memory block may or may not be immediately destroyed andthis behavior may differ across platforms. Attempts to access datainside the shared memory block afterunlink() has been called mayresult in memory access errors. Note: the last process relinquishingits hold on a shared memory block may callunlink() andclose() in either order.

buf¶

A memoryview of contents of the shared memory block.

name¶

Read-only access to the unique name of the shared memory block.

size¶

Read-only access to size in bytes of the shared memory block.

The following example demonstrates low-level use ofSharedMemoryinstances:

>>>frommultiprocessingimportshared_memory>>>shm_a=shared_memory.SharedMemory(create=True,size=10)>>>type(shm_a.buf)<class 'memoryview'>>>>buffer=shm_a.buf>>>len(buffer)10>>>buffer[:4]=bytearray([22,33,44,55])# Modify multiple at once>>>buffer[4]=100# Modify single byte at a time>>># Attach to an existing shared memory block>>>shm_b=shared_memory.SharedMemory(shm_a.name)>>>importarray>>>array.array('b',shm_b.buf[:5])# Copy the data into a new array.arrayarray('b', [22, 33, 44, 55, 100])>>>shm_b.buf[:5]=b'howdy'# Modify via shm_b using bytes>>>bytes(shm_a.buf[:5])# Access via shm_ab'howdy'>>>shm_b.close()# Close each SharedMemory instance>>>shm_a.close()>>>shm_a.unlink()# Call unlink only once to release the shared memory

The following example demonstrates a practical use of theSharedMemoryclass withNumPy arrays, accessing thesamenumpy.ndarray from two distinct Python shells:

>>># In the first Python interactive shell>>>importnumpyasnp>>>a=np.array([1,1,2,3,5,8])# Start with an existing NumPy array>>>frommultiprocessingimportshared_memory>>>shm=shared_memory.SharedMemory(create=True,size=a.nbytes)>>># Now create a NumPy array backed by shared memory>>>b=np.ndarray(a.shape,dtype=a.dtype,buffer=shm.buf)>>>b[:]=a[:]# Copy the original data into shared memory>>>barray([1, 1, 2, 3, 5, 8])>>>type(b)<class 'numpy.ndarray'>>>>type(a)<class 'numpy.ndarray'>>>>shm.name# We did not specify a name so one was chosen for us'psm_21467_46075'>>># In either the same shell or a new Python shell on the same machine>>>importnumpyasnp>>>frommultiprocessingimportshared_memory>>># Attach to the existing shared memory block>>>existing_shm=shared_memory.SharedMemory(name='psm_21467_46075')>>># Note that a.shape is (6,) and a.dtype is np.int64 in this example>>>c=np.ndarray((6,),dtype=np.int64,buffer=existing_shm.buf)>>>carray([1, 1, 2, 3, 5, 8])>>>c[-1]=888>>>carray([  1,   1,   2,   3,   5, 888])>>># Back in the first Python interactive shell, b reflects this change>>>barray([  1,   1,   2,   3,   5, 888])>>># Clean up from within the second Python shell>>>delc# Unnecessary; merely emphasizing the array is no longer used>>>existing_shm.close()>>># Clean up from within the first Python shell>>>delb# Unnecessary; merely emphasizing the array is no longer used>>>shm.close()>>>shm.unlink()# Free and release the shared memory block at the very end

classmultiprocessing.managers.SharedMemoryManager([address[,authkey]])¶

A subclass ofmultiprocessing.managers.BaseManager which can beused for the management of shared memory blocks across processes.

A call tostart() on aSharedMemoryManager instance causes a new process to be started.This new process’s sole purpose is to manage the life cycleof all shared memory blocks created through it. To trigger the releaseof all shared memory blocks managed by that process, callshutdown() on the instance.This triggers aunlink() callon all of theSharedMemory objects managed by that process and thenstops the process itself. By creatingSharedMemory instancesthrough aSharedMemoryManager, we avoid the need to manually trackand trigger the freeing of shared memory resources.

This class provides methods for creating and returningSharedMemoryinstances and for creating a list-like object (ShareableList)backed by shared memory.

Refer toBaseManager for a descriptionof the inheritedaddress andauthkey optional input arguments and howthey may be used to connect to an existingSharedMemoryManager servicefrom other processes.

SharedMemory(size)¶

Create and return a newSharedMemory object with thespecifiedsize in bytes.

ShareableList(sequence)¶

Create and return a newShareableList object, initializedby the values from the inputsequence.

The following example demonstrates the basic mechanisms of aSharedMemoryManager:

>>>frommultiprocessing.managersimportSharedMemoryManager>>>smm=SharedMemoryManager()>>>smm.start()# Start the process that manages the shared memory blocks>>>sl=smm.ShareableList(range(4))>>>slShareableList([0, 1, 2, 3], name='psm_6572_7512')>>>raw_shm=smm.SharedMemory(size=128)>>>another_sl=smm.ShareableList('alpha')>>>another_slShareableList(['a', 'l', 'p', 'h', 'a'], name='psm_6572_12221')>>>smm.shutdown()# Calls unlink() on sl, raw_shm, and another_sl

The following example depicts a potentially more convenient pattern for usingSharedMemoryManager objects via thewith statement to ensure that all shared memory blocks are releasedafter they are no longer needed:

>>>withSharedMemoryManager()assmm:...sl=smm.ShareableList(range(2000))...# Divide the work among two processes, storing partial results in sl...p1=Process(target=do_work,args=(sl,0,1000))...p2=Process(target=do_work,args=(sl,1000,2000))...p1.start()...p2.start()# A multiprocessing.Pool might be more efficient...p1.join()...p2.join()# Wait for all work to complete in both processes...total_result=sum(sl)# Consolidate the partial results now in sl

When using aSharedMemoryManagerin awith statement, the shared memory blocks created using thatmanager are all released when thewith statement’s code blockfinishes execution.

classmultiprocessing.shared_memory.ShareableList(sequence=None,*,name=None)¶

Provide a mutable list-like object where all values stored within arestored in a shared memory block.This constrains storable values to the following built-in data types:

• int (signed 64-bit)

• float

• bool

• str (less than 10M bytes each when encoded as UTF-8)

• bytes (less than 10M bytes each)

• None

It also notably differs from the built-inlist typein that these lists can not change their overall length(i.e. noappend(),insert(), etc.) and do notsupport the dynamic creation of newShareableList instancesvia slicing.

sequence is used in populating a newShareableList full of values.Set toNone to instead attach to an already existingShareableList by its unique shared memory name.

name is the unique name for the requested shared memory, as describedin the definition forSharedMemory. When attaching to anexistingShareableList, specify its shared memory block’s uniquename while leavingsequence set toNone.

Note

A known issue exists forbytes andstr values.If they end with\x00 nul bytes or characters, those may besilently stripped when fetching them by index from theShareableList. This.rstrip(b'\x00') behavior isconsidered a bug and may go away in the future. Seegh-106939.

For applications where rstripping of trailing nulls is a problem,work around it by always unconditionally appending an extra non-0byte to the end of such values when storing and unconditionallyremoving it when fetching:

>>>frommultiprocessingimportshared_memory>>>nul_bug_demo=shared_memory.ShareableList(['?\x00',b'\x03\x02\x01\x00\x00\x00'])>>>nul_bug_demo[0]'?'>>>nul_bug_demo[1]b'\x03\x02\x01'>>>nul_bug_demo.shm.unlink()>>>padded=shared_memory.ShareableList(['?\x00\x07',b'\x03\x02\x01\x00\x00\x00\x07'])>>>padded[0][:-1]'?\x00'>>>padded[1][:-1]b'\x03\x02\x01\x00\x00\x00'>>>padded.shm.unlink()

count(value)¶

Return the number of occurrences ofvalue.

index(value)¶

Return first index position ofvalue.RaiseValueError ifvalue is not present.

format¶

Read-only attribute containing thestruct packing format used byall currently stored values.

shm¶

TheSharedMemory instance where the values are stored.

The following example demonstrates basic use of aShareableListinstance:

>>>frommultiprocessingimportshared_memory>>>a=shared_memory.ShareableList(['howdy',b'HoWdY',-273.154,100,None,True,42])>>>[type(entry)forentryina][<class 'str'>, <class 'bytes'>, <class 'float'>, <class 'int'>, <class 'NoneType'>, <class 'bool'>, <class 'int'>]>>>a[2]-273.154>>>a[2]=-78.5>>>a[2]-78.5>>>a[2]='dry ice'# Changing data types is supported as well>>>a[2]'dry ice'>>>a[2]='larger than previously allocated storage space'Traceback (most recent call last):...ValueError:exceeds available storage for existing str>>>a[2]'dry ice'>>>len(a)7>>>a.index(42)6>>>a.count(b'howdy')0>>>a.count(b'HoWdY')1>>>a.shm.close()>>>a.shm.unlink()>>>dela# Use of a ShareableList after call to unlink() is unsupported

The following example depicts how one, two, or many processes may access thesameShareableList by supplying the name of the shared memory blockbehind it:

>>>b=shared_memory.ShareableList(range(5))# In a first process>>>c=shared_memory.ShareableList(name=b.shm.name)# In a second process>>>cShareableList([0, 1, 2, 3, 4], name='...')>>>c[-1]=-999>>>b[-1]-999>>>b.shm.close()>>>c.shm.close()>>>c.shm.unlink()

The following examples demonstrates thatShareableList(and underlyingSharedMemory) objectscan be pickled and unpickled if needed.Note, that it will still be the same shared object.This happens, because the deserialized object hasthe same unique name and is just attached to an existingobject with the same name (if the object is still alive):

>>>importpickle>>>frommultiprocessingimportshared_memory>>>sl=shared_memory.ShareableList(range(10))>>>list(sl)[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

>>>deserialized_sl=pickle.loads(pickle.dumps(sl))>>>list(deserialized_sl)[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

>>>sl[0]=-1>>>deserialized_sl[1]=-2>>>list(sl)[-1, -2, 2, 3, 4, 5, 6, 7, 8, 9]>>>list(deserialized_sl)[-1, -2, 2, 3, 4, 5, 6, 7, 8, 9]

>>>sl.shm.close()>>>sl.shm.unlink()