Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

bpo-46799: Improve multiprocessing shared_memory ShareableList#31467

Open
tcl326 wants to merge 6 commits intopython:mainfrom
tcl326:fix-issue-46799
Open

bpo-46799: Improve multiprocessing shared_memory ShareableList#31467
tcl326 wants to merge 6 commits intopython:mainfrom
tcl326:fix-issue-46799

Conversation

@tcl326
Copy link

@tcl326tcl326 commentedFeb 21, 2022
edited by bedevere-bot
Loading

This PR improves ShareableList's memory usage and performance by consolidating the metadata of each items into a single area in the shared memory block. This allow us to fetch the metadata needed to read and set item using a singlestruct.unpack_from call. Additionally, this MR also removesself._allocated_offsets the offsets can be obtained directly from the shared memory and there is no need to keep a copy in ShareableList. Further more,self._allocated_offsets can has a significant memory footprint when the number of items in the list is large. I am seeing a significant savings in memory footprint and increase in performance in a benchmark where a ShareableList with a list of 10,000,000 zeros is forked by 5 child processes, and each child precess iterate through the list on their own.

Additionally, this PR also include the fix to#26328, and an additional fix where we properly allocate 8 bytes of memory for a string or byte array of size 8. Previously, we would've allocated 16 bytes for a string of size 8 or byte array of size 8.

Here is the resulting memory usage plotted usinghttps://github.com/pythonprofilers/memory_profiler

  • Memory consumption of the current ShareableList in multiprocessing
    shareablelist

  • Memory consumption of the ShareableList proposed in this MR
    fixedshareablelist

https://bugs.python.org/issue46799

@the-knights-who-say-ni

Hello, and thanks for your contribution!

I'm a bot set up to make sure that the project can legally accept this contribution by verifying everyone involved has signed thePSF contributor agreement (CLA).

CLA Missing

Our records indicate the following people have not signed the CLA:

@tcl326

For legal reasons we need all the people listed to sign the CLA before we can look at your contribution. Please followthe steps outlined in the CPython devguide to rectify this issue.

If you have recently signed the CLA, please wait at least one business day
before our records are updated.

You cancheck yourself to see if the CLA has been received.

Thanks again for the contribution, we look forward to reviewing it!

@hugovkhugovk added the performancePerformance or resource usage labelFeb 21, 2022
@terryjreedyterryjreedy changed the titlebpo-46799: Improve ShareableList memory utilization and Performancebpo-46799: Improve multiprocessing shared_memory ShareableListFeb 25, 2022
@ghost
Copy link

ghost commentedMar 7, 2023
edited by ghost
Loading

All commit authors signed the Contributor License Agreement.
CLA signed

@arhadthedev
Copy link
Member

@tcl326 Could you sign the new CLA by clickingnot signed button in the cpython-cla-bot's message, please?

:class:`multiprocessing.shared_memory.ShareableList` performance by merging
the area in shared memory dedicated to offsets and packing formats together.
This allows a single :func:`struct.unpack_from` call to retrieve both the
offset and the packing format of a sinlge entry Fix UnicodeDecodeError with
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

small typo sinlge

Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

@tirantiranAwaiting requested review from tiran

1 more reviewer

@gstgstgst left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@tcl326@the-knights-who-say-ni@arhadthedev@gst@vstinner@hugovk@ezio-melotti@bedevere-bot

[8]ページ先頭

©2009-2026 Movatter.jp