Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

WIP: gh-114467: Support posix_spawn in subprocess.Popen with cwd != None#114468

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Draft
kulikjak wants to merge8 commits intopython:main
base:main
Choose a base branch
Loading
fromkulikjak:posix_spawn-cwd-support

Conversation

kulikjak
Copy link
Contributor

@kulikjakkulikjak commentedJan 23, 2024
edited by github-actionsbot
Loading

This is a work in progress as there is still one issue to solve:

Runningsubprocess.run(["/usr/bin/ls"], cwd="/nonexistent") should fail with:
FileNotFoundError: [Errno 2] No such file or directory: '/nonexistent'

but with this PR, it fails with:
FileNotFoundError: [Errno 2] No such file or directory: '/usr/bin/ls'

which is incorrect, because it's not '/usr/bin/ls' that is missing, it's the directory. I have yet to determine how to fix this.

(this also causestest.test_subprocess.POSIXProcessTestCase.test_exception_cwd to fail).


📚 Documentation preview 📚:https://cpython-previews--114468.org.readthedocs.build/

@kulikjak
Copy link
ContributorAuthor

Oh, and based on my quick search, I just realized that some systems haveposix_spawn_file_actions_addchdir_np (OpenBSD, glib, MacOS), some haveposix_spawn_file_actions_addchdir (NetBSD), and some have both (Oracle Solaris). That is another thing that needs some work.

@erlend-aaslanderlend-aasland marked this pull request as draftJanuary 23, 2024 09:00
@erlend-aasland
Copy link
Contributor

Marking this as a draft (while it is WIP).

@gpshead
Copy link
Member

Thetest_exception_cwd behavior was added for a recent bugfix. In the case of posix_spawn, I don't believe we can differentiate error causes when a chdir action is used. All we get is the errno but no indication as towhich operation internal to posix_spawn it came from. So for user friendliness we probably want a compromise error message when posix_spawn was used and a cwd was specified on errors that might involve a path that includes both paths in the error message and indicates that we don't have information as to which the error applies to.(while we could attempt to introspect the system and guess, that wouldn't be technically correct due to timing issues and more complicated, so it's better not to do that)

serhiy-storchaka and kulikjak reacted with thumbs up emoji

Copy link
Member

@serhiy-storchakaserhiy-storchaka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Please add an entry in What's New.

@kulikjak
Copy link
ContributorAuthor

Thetest_exception_cwd behavior was added for a recent bugfix. In the case of posix_spawn, I don't believe we can differentiate error causes when a chdir action is used. All we get is the errno but no indication as towhich operation internal to posix_spawn it came from. So for user friendliness we probably want a compromise error message when posix_spawn was used and a cwd was specified on errors that might involve a path that includes both paths in the error message and indicates that we don't have information as to which the error applies to.

Thanks! My experimentation led me to the same conclusion that determining the problematic path might not be feasible. Having both paths in the exception message seems like a good solution; I will look into it.

@kulikjakkulikjakforce-pushed theposix_spawn-cwd-support branch from611a339 to12553e1CompareMarch 24, 2025 15:52
@kulikjak
Copy link
ContributorAuthor

Sorry it took me so long, but I finally found some time to move this further. The exception is now better, although the fact that it's not easily possible to find out whether it's the command orcwd that doesn't exist is IMHO not exactly elegant...

@kulikjak
Copy link
ContributorAuthor

Here is how the exception looks now:

Traceback (most recent call last):  File "/scratch/with.py", line 6, in <module>    res = subprocess.run(["/usr/bin/ls"], cwd="/nonexistent")          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File "/usr/lib/python3.11/subprocess.py", line 548, in run    with Popen(*popenargs, **kwargs) as process:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^  File "/usr/lib/python3.11/subprocess.py", line 1028, in __init__    self._execute_child(args, executable, preexec_fn, close_fds,  File "/usr/lib/python3.11/subprocess.py", line 1846, in _execute_child    self._posix_spawn(args, executable, env, restore_signals,  File "/usr/lib/python3.11/subprocess.py", line 1790, in _posix_spawn    self.pid = os.posix_spawn(executable, args, env, **kwargs)               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^FileNotFoundError: Either '/usr/bin/ls' or '/nonexistent' doesn't exist.

Comment on lines +7407 to +7408
Py_XDECREF(*cwd);
*cwd = path;

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

POSIX_SPAWN_CHDIR can be passed multiple times with different arguments. In thet case the temporary bytes object saved in*cwd can be deallocated, and theposix_spawn_file_actions_addchdir_np() argument can became a bad pointer.

I think that we need to usetemp_buffer here.

Please add a test forposix_spawn() with multiplePOSIX_SPAWN_CHDIR actions.

Copy link
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Yeah, I am not sure how to handle multiplePOSIX_SPAWN_CHDIR actions.

We can either:

  1. callposix_spawn_file_actions_addchdir_np for eachPOSIX_SPAWN_CHDIR in which caseposix_spawn will fail when any of those doesn't exist (or when other error occurs as you pointed out below) even if that one wouldn't be the final one. In such case, the error message might be pretty big.

  2. we can omit all but the last one, in which case it will not behave as some might expect. But we would have only two paths to display in the error message.

As for the bad pointer, it's possible, butcwd always holds the latest one so whether it's called with one or muliple ones shouldn't matter here? But I might be completely wrong. Also, if we want to save all the paths (in case 1. is the prefered one), that might change.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

posix_spawn_file_actions_addchdir_np also affects how files are found for laterposix_spawn_file_actions_add_open, so you need to do them all.

@@ -7518,6 +7543,17 @@ py_posix_spawn(int use_posix_spawnp, PyObject *module, path_t *path, PyObject *a

if (err_code) {
errno = err_code;
#ifdef HAVE_POSIX_SPAWN_FILE_ACTIONS_ADDCHDIR_NP
if (errno == ENOENT && cwd != NULL) {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

This is not the only common error possible here. For example, you can get EPERM or EACCES if any of the executable or the cwd paths are not readable for the user, or ENAMETOOLONG if they are too long, or ELOOP if they contain symlink loops, or ENOTDIR if any of intermediate path component is not a directory. It is not possible to handle all errors.

Maybe usePyErr_SetFromErrnoWithFilenameObjects() (notes at end) with too path arguments? It will perhaps not work if there are multiplePOSIX_SPAWN_CHDIR actions, but this is an extremely uncommon case.

Copy link
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

The problem withPyErr_SetFromErrnoWithFilenameObjects is that the exception looks like this:

  File "/usr/lib/python3.13/subprocess.py", line 1799, in _posix_spawn    self.pid = os.posix_spawn(executable, args, env, **kwargs)               ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^FileNotFoundError: [Errno 2] No such file or directory: '/usr/bin/ls' -> b'/nonexistent'

and I don't like the arrow, which is why I used thePyErr_Format. But otherwise it works nicely and the change is simpler:

-PyErr_SetFromErrnoWithFilenameObject(PyExc_OSError, path->object);+PyErr_SetFromErrnoWithFilenameObjects(PyExc_OSError, path->object, cwd);

MultiplePOSIX_SPAWN_CHDIR actions make it more complicated...

Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment
Reviewers

@dmcookedmcookedmcooke left review comments

@serhiy-storchakaserhiy-storchakaserhiy-storchaka left review comments

@gpsheadgpsheadAwaiting requested review from gpsheadgpshead is a code owner

@erlend-aaslanderlend-aaslandAwaiting requested review from erlend-aaslanderlend-aasland is a code owner

@corona10corona10Awaiting requested review from corona10corona10 is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

5 participants
@kulikjak@erlend-aasland@gpshead@dmcooke@serhiy-storchaka

[8]ページ先頭

©2009-2025 Movatter.jp