Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork34.2k
bpo-46799: Improve multiprocessing shared_memory ShareableList#31467
bpo-46799: Improve multiprocessing shared_memory ShareableList#31467tcl326 wants to merge 6 commits intopython:mainfrom
Conversation
the-knights-who-say-ni commentedFeb 21, 2022
Hello, and thanks for your contribution! I'm a bot set up to make sure that the project can legally accept this contribution by verifying everyone involved has signed thePSF contributor agreement (CLA). CLA MissingOur records indicate the following people have not signed the CLA: For legal reasons we need all the people listed to sign the CLA before we can look at your contribution. Please followthe steps outlined in the CPython devguide to rectify this issue. If you have recently signed the CLA, please wait at least one business day You cancheck yourself to see if the CLA has been received. Thanks again for the contribution, we look forward to reviewing it! |
ghost commentedMar 7, 2023 • edited by ghost
Loading Uh oh!
There was an error while loading.Please reload this page.
edited by ghost
Uh oh!
There was an error while loading.Please reload this page.
arhadthedev commentedApr 2, 2023
@tcl326 Could you sign the new CLA by clicking |
| :class:`multiprocessing.shared_memory.ShareableList` performance by merging | ||
| the area in shared memory dedicated to offsets and packing formats together. | ||
| This allows a single :func:`struct.unpack_from` call to retrieve both the | ||
| offset and the packing format of a sinlge entry Fix UnicodeDecodeError with |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
small typo sinlge
Uh oh!
There was an error while loading.Please reload this page.
This PR improves ShareableList's memory usage and performance by consolidating the metadata of each items into a single area in the shared memory block. This allow us to fetch the metadata needed to read and set item using a single
struct.unpack_fromcall. Additionally, this MR also removesself._allocated_offsetsthe offsets can be obtained directly from the shared memory and there is no need to keep a copy in ShareableList. Further more,self._allocated_offsetscan has a significant memory footprint when the number of items in the list is large. I am seeing a significant savings in memory footprint and increase in performance in a benchmark where a ShareableList with a list of 10,000,000 zeros is forked by 5 child processes, and each child precess iterate through the list on their own.Additionally, this PR also include the fix to#26328, and an additional fix where we properly allocate 8 bytes of memory for a string or byte array of size 8. Previously, we would've allocated 16 bytes for a string of size 8 or byte array of size 8.
Here is the resulting memory usage plotted usinghttps://github.com/pythonprofilers/memory_profiler
Memory consumption of the current ShareableList in multiprocessing

Memory consumption of the ShareableList proposed in this MR

https://bugs.python.org/issue46799