
This issue trackerhas been migrated toGitHub, and is currentlyread-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.
Created on2017-09-07 09:58 byhowarthjw, last changed2022-04-11 14:58 byadmin. This issue is nowclosed.
| Pull Requests | |||
|---|---|---|---|
| URL | Status | Linked | Edit |
| PR 4720 | merged | ned.deily,2017-12-05 04:25 | |
| PR 4721 | merged | python-dev,2017-12-05 04:43 | |
| Messages (9) | |||
|---|---|---|---|
| msg301580 -(view) | Author: Jack Howarth (howarthjw) | Date: 2017-09-07 09:58 | |
The Python 3.x test suite produces a new regression on macOS 10.13 under the new APFS filesystem when executing the test_undecodable_filename() test fromLib/test/test_httpservers.py. The error appears as...======================================================================ERROR: test_undecodable_filename(test.test_httpservers.SimpleHTTPServerTestCase)----------------------------------------------------------------------Traceback (most recent call last): File "/Users/howarth/Python-3.6.2/Lib/test/support/__init__.py",line 601, in wrapper return func(*args, **kw) File "/Users/howarth/Python-3.6.2/Lib/test/test_httpservers.py",line 380, in test_undecodable_filename with open(os.path.join(self.tempdir, filename), 'wb') as f:OSError: [Errno 92] Illegal byte sequence:'/var/folders/7g/1x2rsy3j40n1pydq931hzlkm0000gn/T/tmpomp1r36b/@test_58317_tmp\udce7w\udcf0.txt'----------------------------------------------------------------------Ran 59 tests in 3.207sSeehttps://developer.apple.com/library/content/documentation/FileManagement/Conceptual/APFS_Guide/FAQ/FAQ.html"""APFS accepts only valid UTF-8 encoded filenames for creation, and preserves both case and normalization of the filename on disk in all variants..Some differences between how APFS and HFS+ handle filenames include the following:... • APFS doesn’t allow files to be created with filenames that contain unassigned codepoints in the Unicode 9.0 standard, whereas HFS+ does. | |||
| msg306006 -(view) | Author: Matt Billenstein (mattbillenstein) | Date: 2017-11-10 06:20 | |
Reflected on the High Sierra buildbot now:http://buildbot.python.org/all/#/builders/14/builds/162 | |||
| msg307106 -(view) | Author: Ned Deily (ned.deily)*![]() | Date: 2017-11-28 02:41 | |
More background: APFS became available in macOS 10.12 although not used by default. But with the current macOS 10.13 (High Sierra), root file systems on certain devices, such as SSDs, are automatically migrated from HFS+ to APFS so are much more likely to be seen starting in 10.13 and, in general, there is no easy way of knowing what type of FS a particular file resides on.# TMPDIR on APFS volume$ ./bin/python3.7 -m test test_httpserversRun tests sequentially0:00:00 load avg: 1.43 [1/1] test_httpserverstest test_httpservers failed -- Traceback (most recent call last): File ".../lib/python3.7/test/support/__init__.py", line 598, in wrapper return func(*args, **kw) File ".../lib/python3.7/test/test_httpservers.py", line 395, in test_undecodable_filename with open(os.path.join(self.tempdir, filename), 'wb') as f:OSError: [Errno 92] Illegal byte sequence: '/var/folders/sn/0m4rnbyj2z1byjs68sz838300000gn/T/tmpqeg89k76/@test_64005_tmp\udce7w\udcf0.txt'test_httpservers failed1 test failed: test_httpserversTotal duration: 3 secTests result: FAILURE# TMPDIR redirected to HFS+ volume$ TMPDIR=/Volumes/euterpe/a ./bin/python3.7 -m test test_httpserversRun tests sequentially0:00:00 load avg: 1.28 [1/1] test_httpservers1 test OK.Total duration: 3 secTests result: SUCCESS@vstinner, as the expert in this area, what do you suggest as the best way to deal with this? In support.__init__.py? Or test_httpservers itself? | |||
| msg307109 -(view) | Author: Serhiy Storchaka (serhiy.storchaka)*![]() | Date: 2017-11-28 07:38 | |
I suggest to catch an error and skip the test if OSError with the specific errno has been raised. | |||
| msg307114 -(view) | Author: STINNER Victor (vstinner)*![]() | Date: 2017-11-28 09:54 | |
> @vstinner, as the expert in this area, what do you suggest as the best way to deal with this? In support.__init__.py? Or test_httpservers itself?It's tricky when a filename is valid or not depending on the path, depending on the filesystem. The tempfile module checks if O_TMPFILE is supported: same issue, it depends on the path, and the user can configure the path. So we check each time if O_TMPFILE works or not. | |||
| msg307116 -(view) | Author: Ronald Oussoren (ronaldoussoren)*![]() | Date: 2017-11-28 10:25 | |
Another option is to skip this test unconditionally on macOS when using APFS, the test is already skipped on win32 and tests a generic codepath where only testing on Linux should be fine.BTW. It is fairly easy to detect if a path is on APFS:```import subprocess def is_apfs(path): lines = subprocess.check_output(['df', path]).decode('utf-8').splitlines() mountpoint = lines[1].split(None, 1)[0] lines = subprocess.check_output(['mount']).decode('utf-8').splitlines() for ln in lines: path = ln.split(None, 1)[0] if path == mountpoint: return '(apfs' in ln return False``` | |||
| msg307641 -(view) | Author: Ned Deily (ned.deily)*![]() | Date: 2017-12-05 04:42 | |
New changesetb3edde8dd44c878e9f039a2165d00ff075157d4b by Ned Deily in branch 'master':bpo-31380: Skip test_httpservers test_undecodable_file on macOS. (#4720)https://github.com/python/cpython/commit/b3edde8dd44c878e9f039a2165d00ff075157d4b | |||
| msg307643 -(view) | Author: Ned Deily (ned.deily)*![]() | Date: 2017-12-05 05:03 | |
New changesetd9cadc5f597e5966132c9249f6c7ff0ed1eba0cb by Ned Deily (Miss Islington (bot)) in branch '3.6':[3.6]bpo-31380: Skip test_httpservers test_undecodable_file on macOS. (GH-4720) (#4721)https://github.com/python/cpython/commit/d9cadc5f597e5966132c9249f6c7ff0ed1eba0cb | |||
| msg307644 -(view) | Author: Ned Deily (ned.deily)*![]() | Date: 2017-12-05 05:09 | |
I've chosen to unconditionally skip test_undecodable_filename on macOS; it was already skipped for older macOS systems and I think it would be a mistake to test for a specific error code that could change in later releases. As Ronald points out, testing on Linux should be sufficient. Thanks everyone for your input. Fixed in 3.6.4 and 3.7.0. | |||
| History | |||
|---|---|---|---|
| Date | User | Action | Args |
| 2022-04-11 14:58:52 | admin | set | github: 75561 |
| 2017-12-05 05:09:03 | ned.deily | set | status: open -> closed resolution: fixed messages: +msg307644 stage: patch review -> resolved |
| 2017-12-05 05:03:31 | ned.deily | set | messages: +msg307643 |
| 2017-12-05 04:43:12 | python-dev | set | pull_requests: +pull_request4630 |
| 2017-12-05 04:42:04 | ned.deily | set | messages: +msg307641 |
| 2017-12-05 04:25:25 | ned.deily | set | keywords: +patch stage: patch review pull_requests: +pull_request4629 |
| 2017-11-28 10:25:36 | ronaldoussoren | set | messages: +msg307116 |
| 2017-11-28 09:54:15 | vstinner | set | messages: +msg307114 |
| 2017-11-28 07:38:06 | serhiy.storchaka | set | nosy: +serhiy.storchaka messages: +msg307109 |
| 2017-11-28 02:41:30 | ned.deily | set | messages: +msg307106 |
| 2017-11-17 14:51:49 | vstinner | set | nosy: +vstinner |
| 2017-11-17 14:51:44 | vstinner | link | issue32061 superseder |
| 2017-11-10 09:46:45 | serhiy.storchaka | set | type: crash -> behavior components: + Tests versions: - Python 3.3, Python 3.4, Python 3.5 |
| 2017-11-10 09:45:27 | serhiy.storchaka | link | issue32000 superseder |
| 2017-11-10 06:20:37 | mattbillenstein | set | nosy: +mattbillenstein messages: +msg306006 |
| 2017-09-07 09:58:29 | howarthjw | create | |