- Notifications
You must be signed in to change notification settings - Fork7.7k
Fix Start-Process -Wait polling#24711
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Uh oh!
There was an error while loading.Please reload this page.
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Only style comments.
Also please add more info in the PR description: motivation, links to blog posts.
src/Microsoft.PowerShell.Commands.Management/commands/management/Process.cs OutdatedShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
src/Microsoft.PowerShell.Commands.Management/commands/management/Process.cs OutdatedShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
out completionCode, | ||
out _, | ||
out _, | ||
INFINITE); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
INFINITE
Will does Ctrl-C still kill the cmdlet/pipeline?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
That’s a good point, I never hooked that up.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
This SO post contains a reasonable-looking summary of possible ways to wake up the thread:https://stackoverflow.com/questions/32921513/how-to-force-getqueuedcompletionstatus-to-return-immediately
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Given that we only have a single thread on the completion port, I think the cleanest option is to usePostQueuedCompletionStatus
and post a custom completion to the IOCP.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
I've added code to post the sameJOB_OBJECT_MSG_ACTIVE_PROCESS_ZERO
completion code when a cancellation is done. This means we don't step on any other completion codes added in the future and a simple is cancelled check means we can differentiate between no processes and cancelled.
src/Microsoft.PowerShell.Commands.Management/commands/management/Process.cs OutdatedShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
int dwSize = 0; | ||
const int JOB_OBJECT_BASIC_PROCESS_ID_LIST = 3; | ||
JOBOBJECT_BASIC_PROCESS_ID_LIST JobList = new(); | ||
if (this._completionPortHandle == nint.Zero) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Any specific reason not to just use== 0
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Just personal preference, I prefer to be explicit that we are working with native int types with these checks. The generated IL for both methods is practically the same.
@@ -9,7 +9,7 @@ internal static partial class Interop | |||
{ | |||
internal static partial class Windows | |||
{ | |||
[LibraryImport("Kernel32.dll", SetLastError = true)] | |||
[LibraryImport("api-ms-win-core-job-l2-1-0.dll", SetLastError = true)] |
MatejKafkaDec 27, 2024 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Are you sure we should depend on the specific API set? Documentation recommends to link with kernel32.dll, and CsWin32 also uses kernel32.dll, so I'd follow that convention.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Good point, I'll update them to use only what is documented.
jborean93 commentedJan 4, 2025 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
@iSazonov someone should look at why code factor is not respecting the rules set by PowerShell. You can see the failures from missing This problem has existed all the way back in September, see6736622 for a change I made to satisfy code factor and
|
@jborean93 I filtered out two noise issues on CodeFactor site. It seems now they takes into account only .editorconfig file. At least their documentation points only to this file. I will open a new issue here so that someone migrates the Settings.StyleCop rules to this file. Also Code Factor check isoptional. Usually maintainers ask contributors to fix some CodeFactor issues. Allmandatory rules are in .globalconfig file and violations of the rules will break the build forcing contributor to fix the code. |
src/Microsoft.PowerShell.Commands.Management/commands/management/JobProcessCollection.cs OutdatedShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
src/Microsoft.PowerShell.Commands.Management/commands/management/JobProcessCollection.cs OutdatedShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
src/Microsoft.PowerShell.Commands.Management/commands/management/JobProcessCollection.cs OutdatedShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
src/Microsoft.PowerShell.Commands.Management/commands/management/JobProcessCollection.cs OutdatedShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
src/Microsoft.PowerShell.Commands.Management/commands/management/JobProcessCollection.cs OutdatedShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
src/Microsoft.PowerShell.Commands.Management/commands/management/Process.csShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
src/Microsoft.PowerShell.Commands.Management/commands/management/Process.csShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
src/System.Management.Automation/engine/Interop/Windows/CreateIoCompletionPort.cs OutdatedShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
src/System.Management.Automation/engine/Interop/Windows/CreateJobObject.cs OutdatedShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
src/System.Management.Automation/engine/Interop/Windows/SetInformationJobObject.cs OutdatedShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
Thanks for looking into that. If this is optional then really the check should be removed. While this isn't probably the best place to talk about it having what looks to be a failing CI check can be annoying for contributors trying to get their PR in a ready for review state. An optional check is ultimately not a check at all and is extra noise and work for everyone involved. It should either be enforced or just removed altogether. As for the other comments, I'll push a commit too add the safe handles back in. I've replied to some of the other review comments you've made though as to why I think they shouldn't change. Appreciate you looking through this. |
@jborean93 Before we continue the code review in details I want to share some conceptual thoughts.
We can not remove CodeFactor because it does useful work for us. For example, look issues for XML documentation comments. Historically pwsh has thousands of empty XML doc comments. If we make CodeFactor mandatory we block whole project since it is not really fix all such issues. We can not disable such issues because we want contributors add XML doc comments in new code. So CodeFactor as "optional" check is acompromise. Our workflow is quite gentle - if the maintainer sees significant CodeFactor notices, it will ask contributor to fix them to follow the repository rules. Looking on a piece of the method we change here, it is conceptually as simple as:
But the real code, due to the previously added workarounds, has become more confusing, and with the current fixes, the situation has become even worse. Another amazing thing is the comment: PowerShell/src/Microsoft.PowerShell.Commands.Management/commands/management/Process.cs Lines 2093 to 2100 inc0d7fb7
For some reason, this comment is located after the code it refers to. Next amazing thing is: PowerShell/src/Microsoft.PowerShell.Commands.Management/commands/management/Process.cs Lines 2133 to 2150 inc0d7fb7
The process is created by constructor (throw on error) and cannot be null. So no need to check this and write terminating error (afterif (Wait.IsPresent) we do the same too).Perhaps I didn't see all the problems when I skimmed through this method. And I wouldn't write all this if you had already fixed this code more than once and probably would again. So I would welcome converting this code into something cleaner and more maintainable. |
jborean93 commentedJan 6, 2025 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
While yes the current code is a bit of a mess I'm not sure I agree that this makes it worse. This change
Yes some work should be done to tidy the code up and probably unpick the attempts to share some of the code flow into their own specific paths but I don't think it should be done in the same PR. I'm happy to open another PR to try and clean some more stuff up later on though.
We've fixed similar code but not this issue, this specific issue around jobs polling every 1 second has always been around.
The comment is directly for the code just below it if(Wait){jobObject=new();jobAssigned=jobObject.AssignProcessToJobObject(processInfo.Process);} It is referring to the fact that Processprocess=Process.GetProcessById(processInfo.ProcessId);jobObject.AssignProcessToJobObject(process.SafeHandle); This was problematic because the handle returned by
I've not looked at this code and think it's unrelated to the problem at hand.
I certainly agree that this should be fixed but I don't think it should be rewritten at the same time as fixing an unrelated problem. Refactoring should only happen when you want to cleanup the code, not do bugfixes at the same time. Doing both at the same time is makes it harder for reviews as it has to deal with both behaviour changes and just general code changes. Doing them in smaller steps makes it easier on both the person implementing the change and the people who review the code. |
/azp run |
Azure Pipelines successfully started running 5 pipeline(s). |
src/Microsoft.PowerShell.Commands.Management/commands/management/JobProcessCollection.cs OutdatedShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
src/Microsoft.PowerShell.Commands.Management/commands/management/JobProcessCollection.cs OutdatedShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
src/Microsoft.PowerShell.Commands.Management/commands/management/JobProcessCollection.csShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
src/Microsoft.PowerShell.Commands.Management/commands/management/JobProcessCollection.csShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
src/Microsoft.PowerShell.Commands.Management/commands/management/JobProcessCollection.cs OutdatedShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
src/Microsoft.PowerShell.Commands.Management/commands/management/JobProcessCollection.cs OutdatedShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
Update the logic for waiting for a process and its dependencies to use acompletion port rather than a spin lock that polls every second. Thismakes it more efficient to wait for it to complete as it will returnstraight away.
Co-authored-by: Ilya <darpa@yandex.ru>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
@jborean93 LGTM with same style comments.
Please don't rebase without maintainer ask. Rebase kill commit history and resets previous reviews.
src/Microsoft.PowerShell.Commands.Management/commands/management/JobProcessCollection.cs OutdatedShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
src/Microsoft.PowerShell.Commands.Management/commands/management/JobProcessCollection.cs OutdatedShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
src/System.Management.Automation/engine/Interop/Windows/CreateIoCompletionPort.cs OutdatedShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
src/System.Management.Automation/engine/Interop/Windows/CreateJobObject.cs OutdatedShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
{ | ||
Interop.Windows.GetQueuedCompletionStatus( | ||
_completionPort, | ||
INFINITE, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Please remove the const and move it to the helper.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Done
CI says that the branch is out of date and I rebase to bring it back up to date. Using a merge pull creates merge commits and while yes a rebase requires a force push it still preserves the commit history for the PR. If you have a better suggestion then please share, personally I'm getting sick of PowerShell's CI that marks PRs as red or not ready when some things seem to be optional. I still see the reviews, I marked them as resolved because I responded to the latest comments on there, sorry if I missed anything. |
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
Uh oh!
There was an error while loading.Please reload this page.
Co-authored-by: Ilya <darpa@yandex.ru>
@jborean93 MSFT team use a tool for reviewingcommit by commit and they ask us to keep commit history. GitHub Web GUI also doesn't like rebase - it can hide some comments (and it is very tricky to find them). "Update branch with merge commit" button is not useful for us since we always do "Rebase and Merge" finally. I have not permission to remove the option. "Update branch with rebase" could be useful for us (mainly maintainers) if we need to rebase for running CIs before merge of very old PR. |
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as duplicate.
This comment was marked as duplicate.
4e79421
intoPowerShell:masterUh oh!
There was an error while loading.Please reload this page.
microsoft-github-policy-servicebot commentedJan 28, 2025 • edited by unfurl-linksbot
Loading Uh oh!
There was an error while loading.Please reload this page.
edited by unfurl-linksbot
Uh oh!
There was an error while loading.Please reload this page.
📣 Hey@jborean93, how did we do? We would love to hear your feedback with the link below! 🗣️ 🔗https://aka.ms/PSRepoFeedback |
Thanks for the review and merge. |
@jborean93 Thanks for great work! I hope you will find the time to clean up the file(s) so that it will be easier for us to do the subsequent work. |
Uh oh!
There was an error while loading.Please reload this page.
PR Summary
Update the logic for waiting for a process and its dependencies to use a completion port rather than a spin lock that polls every second. This makes it more efficient to wait for it to complete as it will return straight away.
Also adds stopping support for Unix and
-Wait
and when a job isn't used to track processes as currently they do not respond toStopProcessing
.PR Context
Fixes:#24709
The new approach is the one recommended by Raymond Chen athttps://devblogs.microsoft.com/oldnewthing/20130405-00/?p=4743. It sets up an IO completion port and monitors the completion codes returned by the job until we've received
JOB_OBJECT_MSG_ACTIVE_PROCESS_ZERO
signalling all the processes have ended inside the job.There should not be any breaking changes here, just an improvement in how we poll for the job completion.
PR Checklist
.h
,.cpp
,.cs
,.ps1
and.psm1
files have the correct copyright headerWIP:
or[ WIP ]
to the beginning of the title (theWIP
bot will keep its status check atPending
while the prefix is present) and remove the prefix when the PR is ready.- [ ] Issue filed:
(which runs in a different PS Host).