Movatterモバイル変換


[0]ホーム

URL:


homepage

Issue40014

This issue trackerhas been migrated toGitHub, and is currentlyread-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title:os.getgrouplist() can fail on macOS if the user has more than 17 groups
Type:enhancementStage:resolved
Components:macOS, TestsVersions:Python 3.9, Python 3.8, Python 3.7
process
Status:closedResolution:fixed
Dependencies:Superseder:
Assigned To:Nosy List: corona10, miss-islington, ned.deily, ronaldoussoren, shihai1991, steve.dower, vstinner
Priority:normalKeywords:patch

Created on2020-03-19 16:04 bycorona10, last changed2022-04-11 14:59 byadmin. This issue is nowclosed.

Pull Requests
URLStatusLinkedEdit
PR 19075closedcorona10,2020-03-19 16:13
PR 19117closedvstinner,2020-03-23 16:12
PR 19118mergedvstinner,2020-03-23 16:56
PR 19123mergedmiss-islington,2020-03-23 19:01
PR 19124mergedmiss-islington,2020-03-23 19:01
PR 19126mergedvstinner,2020-03-23 22:13
PR 19143mergedmiss-islington,2020-03-24 17:22
PR 19144mergedmiss-islington,2020-03-24 17:22
Messages (17)
msg364607 -(view)Author: Dong-hee Na (corona10)*(Python committer)Date: 2020-03-19 16:04
example:https://github.com/python/cpython/pull/19073/checks?check_run_id=519539592I suggest to not to add information for os.getgrouplistif the OSError is raised.
msg364608 -(view)Author: Dong-hee Na (corona10)*(Python committer)Date: 2020-03-19 16:06
or adding <OSError> might be great just like getpwuid
msg364610 -(view)Author: STINNER Victor (vstinner)*(Python committer)Date: 2020-03-19 16:36
These errors are really strange.Steve Dower: are you aware of anything recent change on the macOS job of Azure Pipelines / GH Actions.pythoninfo:---os.uname: posix.uname_result(sysname='Darwin', nodename='Mac-1422.local', release='19.3.0', version='Darwin Kernel Version 19.3.0: Thu Jan  9 20:58:23 PST 2020; root:xnu-6153.81.5~1/RELEASE_X86_64', machine='x86_64')platform.architecture: 64bitplatform.platform: macOS-10.15.3-x86_64-i386-64bit---> example:https://github.com/python/cpython/pull/19073/checks?check_run_id=519539592./python.exe -m test.pythoninfoERROR: collect_pwd() failedTraceback (most recent call last):  File "/Users/runner/runners/2.165.2/work/cpython/cpython/Lib/test/pythoninfo.py", line 761, in collect_info    collect_func(info_add)  File "/Users/runner/runners/2.165.2/work/cpython/cpython/Lib/test/pythoninfo.py", line 336, in collect_pwd    groups = os.getgrouplist(entry.pw_name, entry.pw_gid)OSError: [Errno 0] ErrorCollection failed: exit with error>PR 19075test_posix.test_getgrouplist() failed with:======================================================================ERROR: test_getgrouplist (test.test_posix.PosixTester)----------------------------------------------------------------------Traceback (most recent call last):  File "/Users/runner/runners/2.165.2/work/cpython/cpython/Lib/test/test_posix.py", line 1028, in test_getgrouplist    self.assertIn(group, posix.getgrouplist(user, group))OSError: [Errno 25] Inappropriate ioctl for device
msg364615 -(view)Author: Ned Deily (ned.deily)*(Python committer)Date: 2020-03-19 17:09
This error has to do with the number of groups a particular user is associated with. We’ve squashed similar bugs in the past but it looks like something has changed again in recent releases of macOS. It’s not Azure specific.
msg364862 -(view)Author: STINNER Victor (vstinner)*(Python committer)Date: 2020-03-23 16:58
tl; dr os.getgrouplist() is limited to 17 groups and fails with a random OSError if the request user has more groups.PR 19118 fix the issue.--On macOS-10.15.1-x86_64-i386-64bit (python -m platform), os.getgrouplist() fails with my user name and group identifier:>>> os.getgrouplist('haypo', 20)Traceback (most recent call last):  File "<stdin>", line 1, in <module>OSError: [Errno 0] ErrorFull example:macbook:master haypo$ ./python.exe Python 3.9.0a4+ (heads/master:da2914d, Mar 20 2020, 09:45:36) [Clang 11.0.0 (clang-1100.0.33.8)] on darwinType "help", "copyright", "credits" or "license" for more information.>>> import os, pwd>>> entry=pwd.getpwuid(os.getuid())>>> os.getgrouplist(entry.pw_name, entry.pw_gid)Traceback (most recent call last):  File "<stdin>", line 1, in <module>InterruptedError: [Errno 4] Interrupted system call>>> os.getgrouplist(entry.pw_name, entry.pw_gid)Traceback (most recent call last):  File "<stdin>", line 1, in <module>OSError: [Errno 0] Error>>> entrypwd.struct_passwd(pw_name='haypo', pw_passwd='********', pw_uid=502, pw_gid=20, pw_gecos='Victor Stinner', pw_dir='/Users/haypo', pw_shell='/bin/bash')My user has the following groups:macbook:master haypo$ iduid=502(haypo) gid=20(staff) groups=20(staff),702(com.apple.sharepoint.group.2),12(everyone),61(localaccounts),79(_appserverusr),80(admin),81(_appserveradm),98(_lpadmin),701(com.apple.sharepoint.group.1),33(_appstore),100(_lpoperator),204(_developer),250(_analyticsusers),395(com.apple.access_ftp),398(com.apple.access_screensharing),399(com.apple.access_ssh),400(com.apple.access_remote_ae),703(com.apple.sharepoint.group.3)My user has 18 groups.MAX_GROUPS=16, Python uses 1 + MAX_GROUPS.getgrouplist() manual page says:RETURN VALUES     The getgrouplist() function returns 0 on success.  If the size of the group list is too small to hold     all the user's groups, getgrouplist() returns -1 to indicate failure.  In this case, the group array     will be filled with as many groups as will fit.In short, getgrouplist() doesn't set errno on failure.
msg364864 -(view)Author: STINNER Victor (vstinner)*(Python committer)Date: 2020-03-23 17:22
Python 3.7 and 3.8 are also affected on macOS 10.15 (Catalina).
msg364870 -(view)Author: Ned Deily (ned.deily)*(Python committer)Date: 2020-03-23 18:14
Thanks for the PR, Victor, it does address the problem.  I went back and explicitly tested on macOS 10.14 and 10.13 by creating some more groups and adding the user to them and the same failure occurred there so this is not just a 10.15 issue.  Most likely this hasn't been an issue before is that 10.15 creates more groups for system services than the earlier versions did.  So the PR should be updated to remove the 10.15 (and 10.5!) references.
msg364880 -(view)Author: STINNER Victor (vstinner)*(Python committer)Date: 2020-03-23 18:46
> Most likely this hasn't been an issue before is that 10.15 creates more groups for system services than the earlier versions did.Aha, interesting. I updated my PR to remove mentions of the macOS version number. I just wrote "on macOS". Yeah, I wouldn't be surprised that the function always failed.--I'm not sure that the Linux implementation is correct. It rely on errno, whereas Linux manual page of getgrouplist() doesn't say anything about errno being set on error.Moreover, on Linux, getgrouplist() can also fail with -1 if the group list is too small. In that case, ngroups is updated to the number of groups and so can be used to enlarge the list!
msg364884 -(view)Author: STINNER Victor (vstinner)*(Python committer)Date: 2020-03-23 19:00
test.pythoninfo of the macOS job on the PR:os.getgrouplist: 20, 0, 12, 61, 79, 80, 81, 98, 264, 399, 701, 33, 100, 204, 250, 395, 398, 400os.getgroups: 20, 0, 12, 61, 79, 80, 81, 98, 264, 399, 701, 33, 100, 204, 250, 395, 398, 400So yeah, there are 18 groups ;-)
msg364885 -(view)Author: STINNER Victor (vstinner)*(Python committer)Date: 2020-03-23 19:01
New changeset8ec7370c89aa522602eb9604086ce9f09770953d by Victor Stinner in branch 'master':bpo-40014: Fix os.getgrouplist() on macOS (GH-19118)https://github.com/python/cpython/commit/8ec7370c89aa522602eb9604086ce9f09770953d
msg364888 -(view)Author: miss-islington (miss-islington)Date: 2020-03-23 19:18
New changeset21bee0bd71e1ad270274499f9f58194ebb52e236 by Miss Islington (bot) in branch '3.8':bpo-40014: Fix os.getgrouplist() on macOS (GH-19118)https://github.com/python/cpython/commit/21bee0bd71e1ad270274499f9f58194ebb52e236
msg364889 -(view)Author: miss-islington (miss-islington)Date: 2020-03-23 19:21
New changeset1cdc61c7673f71f2cef57715e482c84efda6d9e0 by Miss Islington (bot) in branch '3.7':bpo-40014: Fix os.getgrouplist() on macOS (GH-19118)https://github.com/python/cpython/commit/1cdc61c7673f71f2cef57715e482c84efda6d9e0
msg364904 -(view)Author: STINNER Victor (vstinner)*(Python committer)Date: 2020-03-23 22:17
> Moreover, on Linux, getgrouplist() can also fail with -1 if the group list is too small. In that case, ngroups is updated to the number of groups and so can be used to enlarge the list!I tested by manually initializing ngroups to a value way lower than MAX_GROUPS (65536 on my Fedora 31): os.getgrouplist() raises OSError() with a random errno, getgrouplist() doesn't set errno on failure.I wrotePR 19126 to fix the issue on all platforms. The glibc implementation is detected and used: use updated ngroups value (only if it's larger).
msg364950 -(view)Author: STINNER Victor (vstinner)*(Python committer)Date: 2020-03-24 17:22
New changesetf5c7cabb2be4e42a5975ba8aac8bb458c8d9d6d7 by Victor Stinner in branch 'master':bpo-40014: Fix os.getgrouplist() (GH-19126)https://github.com/python/cpython/commit/f5c7cabb2be4e42a5975ba8aac8bb458c8d9d6d7
msg364955 -(view)Author: miss-islington (miss-islington)Date: 2020-03-24 17:39
New changeset5753fc69977bd9f70ecb4d466bda650efccf9e0a by Miss Islington (bot) in branch '3.7':bpo-40014: Fix os.getgrouplist() (GH-19126)https://github.com/python/cpython/commit/5753fc69977bd9f70ecb4d466bda650efccf9e0a
msg364956 -(view)Author: miss-islington (miss-islington)Date: 2020-03-24 17:40
New changesetaf6fd1faa68f57c11c862624798f8510b7cac68a by Miss Islington (bot) in branch '3.8':bpo-40014: Fix os.getgrouplist() (GH-19126)https://github.com/python/cpython/commit/af6fd1faa68f57c11c862624798f8510b7cac68a
msg364957 -(view)Author: STINNER Victor (vstinner)*(Python committer)Date: 2020-03-24 17:51
Ned: so Linux was also impacted, but Linux only has an issue with more than 65536 groups :-D macOS MAX_GROUPS is now only 16 (Python uses MAX_GROUPS+1)! Maybe MAX_GROUPS was reduced recently. I'm not sure.Anyway, os.getgrouplist() does no longer depend on MAX_GROUPS hardcoded limit, but grow the group list dynamically. It should now work with any number of groups an all platforms (which provide the getgrouplist() function ;-)) on Python 3.7, 3.8 and master branches.Thanks Dong-hee Na for the bug report and your PR.
History
DateUserActionArgs
2022-04-11 14:59:28adminsetgithub: 84195
2020-03-24 17:51:27vstinnersetstatus: open -> closed
resolution: fixed
messages: +msg364957

stage: patch review -> resolved
2020-03-24 17:40:35miss-islingtonsetmessages: +msg364956
2020-03-24 17:39:21miss-islingtonsetmessages: +msg364955
2020-03-24 17:22:31miss-islingtonsetpull_requests: +pull_request18505
2020-03-24 17:22:23miss-islingtonsetpull_requests: +pull_request18504
2020-03-24 17:22:16vstinnersetmessages: +msg364950
2020-03-23 22:17:06vstinnersetmessages: +msg364904
2020-03-23 22:13:43vstinnersetpull_requests: +pull_request18487
2020-03-23 19:21:14miss-islingtonsetmessages: +msg364889
2020-03-23 19:18:48miss-islingtonsetmessages: +msg364888
2020-03-23 19:01:23miss-islingtonsetpull_requests: +pull_request18485
2020-03-23 19:01:16miss-islingtonsetnosy: +miss-islington
pull_requests: +pull_request18484
2020-03-23 19:01:05vstinnersetmessages: +msg364885
2020-03-23 19:00:15vstinnersetmessages: +msg364884
2020-03-23 18:46:03vstinnersetmessages: +msg364880
2020-03-23 18:14:50ned.deilysetmessages: +msg364870
title: os.getgrouplist() fails on macOS 10.15 (Catalina) if the user has more than 17 groups -> os.getgrouplist() can fail on macOS if the user has more than 17 groups
2020-03-23 17:22:07vstinnersetmessages: +msg364864
versions: + Python 3.7, Python 3.8
2020-03-23 16:59:57vstinnersettitle: os.getgrouplist() can fail on macOS -> os.getgrouplist() fails on macOS 10.15 (Catalina) if the user has more than 17 groups
2020-03-23 16:58:20vstinnersetmessages: +msg364862
2020-03-23 16:56:12vstinnersetpull_requests: +pull_request18479
2020-03-23 16:12:15vstinnersetpull_requests: +pull_request18478
2020-03-22 09:28:19shihai1991setnosy: +shihai1991
2020-03-19 17:11:29ned.deilysettitle: os.getgrouplist() fails on macOS of GH Actions (Azure) -> os.getgrouplist() can fail on macOS
2020-03-19 17:09:33ned.deilysetmessages: +msg364615
2020-03-19 16:36:37vstinnersettitle: os.getgrouplist can raise OSError during the Display build info -> os.getgrouplist() fails on macOS of GH Actions (Azure)
nosy: +ronaldoussoren,ned.deily,steve.dower

messages: +msg364610

versions: + Python 3.9
components: + macOS
2020-03-19 16:13:45corona10setkeywords: +patch
stage: patch review
pull_requests: +pull_request18431
2020-03-19 16:06:14corona10setmessages: +msg364608
2020-03-19 16:04:54corona10create
Supported byThe Python Software Foundation,
Powered byRoundup
Copyright © 1990-2022,Python Software Foundation
Legal Statements

[8]ページ先頭

©2009-2026 Movatter.jp