Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

TarFile filters fail in non-UTF-8 locales #133890

Closed
Assignees
serhiy-storchaka
Labels
3.12only security fixes3.13bugs and security fixes3.14bugs and security fixesstdlibPython modules in the Lib dirtype-bugAn unexpected behavior, bug, or error
@serhiy-storchaka

Description

@serhiy-storchaka

Bug report

test_tarfile files in non-UTF-8 locales. For example:

$ LC_ALL=uk_UA ./python -m test -vuall test_tarfile -m 'NoneInfoExtractTests_*' -m test_data_filter -m test_tar_filter
======================================================================ERROR: setUpClass (test.test_tarfile.NoneInfoExtractTests_Data)----------------------------------------------------------------------Traceback (most recent call last):  File "/home/serhiy/py/cpython/Lib/test/test_tarfile.py", line 3264, in setUpClass    tar.extractall(cls.control_dir, filter=cls.extraction_filter)    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File "/home/serhiy/py/cpython/Lib/tarfile.py", line 2389, in extractall    tarinfo = self._get_extract_tarinfo(member, filter_function, path)  File "/home/serhiy/py/cpython/Lib/tarfile.py", line 2441, in _get_extract_tarinfo    tarinfo = filter_function(tarinfo, path)  File "/home/serhiy/py/cpython/Lib/tarfile.py", line 842, in data_filter    new_attrs = _get_filtered_attrs(member, dest_path, True)  File "/home/serhiy/py/cpython/Lib/tarfile.py", line 782, in _get_filtered_attrs    target_path = os.path.realpath(os.path.join(dest_path, name))  File "/home/serhiy/py/cpython/Lib/posixpath.py", line 405, in realpath    return _realpath(filename, strict, sep, curdir, pardir, getcwd)  File "/home/serhiy/py/cpython/Lib/posixpath.py", line 452, in _realpath    st_mode = lstat(newpath).st_mode              ~~~~~^^^^^^^^^  File "/home/serhiy/py/cpython/Lib/encodings/koi8_u.py", line 12, in encode    return codecs.charmap_encode(input,errors,encoding_table)           ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^UnicodeEncodeError: 'charmap' codec can't encode characters in position 112-118: character maps to <undefined>encoding with 'koi8-u' codec failed======================================================================ERROR: setUpClass (test.test_tarfile.NoneInfoExtractTests_Default)----------------------------------------------------------------------Traceback (most recent call last):  File "/home/serhiy/py/cpython/Lib/test/test_tarfile.py", line 3264, in setUpClass    tar.extractall(cls.control_dir, filter=cls.extraction_filter)    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File "/home/serhiy/py/cpython/Lib/tarfile.py", line 2389, in extractall    tarinfo = self._get_extract_tarinfo(member, filter_function, path)  File "/home/serhiy/py/cpython/Lib/tarfile.py", line 2441, in _get_extract_tarinfo    tarinfo = filter_function(tarinfo, path)  File "/home/serhiy/py/cpython/Lib/tarfile.py", line 842, in data_filter    new_attrs = _get_filtered_attrs(member, dest_path, True)  File "/home/serhiy/py/cpython/Lib/tarfile.py", line 782, in _get_filtered_attrs    target_path = os.path.realpath(os.path.join(dest_path, name))  File "/home/serhiy/py/cpython/Lib/posixpath.py", line 405, in realpath    return _realpath(filename, strict, sep, curdir, pardir, getcwd)  File "/home/serhiy/py/cpython/Lib/posixpath.py", line 452, in _realpath    st_mode = lstat(newpath).st_mode              ~~~~~^^^^^^^^^  File "/home/serhiy/py/cpython/Lib/encodings/koi8_u.py", line 12, in encode    return codecs.charmap_encode(input,errors,encoding_table)           ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^UnicodeEncodeError: 'charmap' codec can't encode characters in position 112-118: character maps to <undefined>encoding with 'koi8-u' codec failed======================================================================ERROR: setUpClass (test.test_tarfile.NoneInfoExtractTests_FullyTrusted)----------------------------------------------------------------------Traceback (most recent call last):  File "/home/serhiy/py/cpython/Lib/test/test_tarfile.py", line 3264, in setUpClass    tar.extractall(cls.control_dir, filter=cls.extraction_filter)    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File "/home/serhiy/py/cpython/Lib/tarfile.py", line 2397, in extractall    self._extract_one(tarinfo, path, set_attrs=not tarinfo.isdir(),    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                      numeric_owner=numeric_owner)                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File "/home/serhiy/py/cpython/Lib/tarfile.py", line 2460, in _extract_one    self._extract_member(tarinfo, os.path.join(path, tarinfo.name),    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                         set_attrs=set_attrs,                         ^^^^^^^^^^^^^^^^^^^^                         numeric_owner=numeric_owner)                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File "/home/serhiy/py/cpython/Lib/tarfile.py", line 2543, in _extract_member    self.makefile(tarinfo, targetpath)    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^  File "/home/serhiy/py/cpython/Lib/tarfile.py", line 2589, in makefile    with bltn_open(targetpath, "wb") as target:         ~~~~~~~~~^^^^^^^^^^^^^^^^^^  File "/home/serhiy/py/cpython/Lib/encodings/koi8_u.py", line 12, in encode    return codecs.charmap_encode(input,errors,encoding_table)           ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^UnicodeEncodeError: 'charmap' codec can't encode characters in position 112-118: character maps to <undefined>encoding with 'koi8-u' codec failed======================================================================ERROR: setUpClass (test.test_tarfile.NoneInfoExtractTests_Tar)----------------------------------------------------------------------Traceback (most recent call last):  File "/home/serhiy/py/cpython/Lib/test/test_tarfile.py", line 3264, in setUpClass    tar.extractall(cls.control_dir, filter=cls.extraction_filter)    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File "/home/serhiy/py/cpython/Lib/tarfile.py", line 2389, in extractall    tarinfo = self._get_extract_tarinfo(member, filter_function, path)  File "/home/serhiy/py/cpython/Lib/tarfile.py", line 2441, in _get_extract_tarinfo    tarinfo = filter_function(tarinfo, path)  File "/home/serhiy/py/cpython/Lib/tarfile.py", line 836, in tar_filter    new_attrs = _get_filtered_attrs(member, dest_path, False)  File "/home/serhiy/py/cpython/Lib/tarfile.py", line 782, in _get_filtered_attrs    target_path = os.path.realpath(os.path.join(dest_path, name))  File "/home/serhiy/py/cpython/Lib/posixpath.py", line 405, in realpath    return _realpath(filename, strict, sep, curdir, pardir, getcwd)  File "/home/serhiy/py/cpython/Lib/posixpath.py", line 452, in _realpath    st_mode = lstat(newpath).st_mode              ~~~~~^^^^^^^^^  File "/home/serhiy/py/cpython/Lib/encodings/koi8_u.py", line 12, in encode    return codecs.charmap_encode(input,errors,encoding_table)           ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^UnicodeEncodeError: 'charmap' codec can't encode characters in position 112-118: character maps to <undefined>encoding with 'koi8-u' codec failed======================================================================ERROR: test_data_filter (test.test_tarfile.TestExtractionFilters.test_data_filter)----------------------------------------------------------------------Traceback (most recent call last):  File "/home/serhiy/py/cpython/Lib/test/test_tarfile.py", line 4086, in test_data_filter    filtered = tarfile.data_filter(tarinfo, '')  File "/home/serhiy/py/cpython/Lib/tarfile.py", line 842, in data_filter    new_attrs = _get_filtered_attrs(member, dest_path, True)  File "/home/serhiy/py/cpython/Lib/tarfile.py", line 782, in _get_filtered_attrs    target_path = os.path.realpath(os.path.join(dest_path, name))  File "/home/serhiy/py/cpython/Lib/posixpath.py", line 405, in realpath    return _realpath(filename, strict, sep, curdir, pardir, getcwd)  File "/home/serhiy/py/cpython/Lib/posixpath.py", line 452, in _realpath    st_mode = lstat(newpath).st_mode              ~~~~~^^^^^^^^^  File "/home/serhiy/py/cpython/Lib/encodings/koi8_u.py", line 12, in encode    return codecs.charmap_encode(input,errors,encoding_table)           ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^UnicodeEncodeError: 'charmap' codec can't encode characters in position 69-75: character maps to <undefined>encoding with 'koi8-u' codec failed======================================================================ERROR: test_tar_filter (test.test_tarfile.TestExtractionFilters.test_tar_filter)----------------------------------------------------------------------Traceback (most recent call last):  File "/home/serhiy/py/cpython/Lib/test/test_tarfile.py", line 4076, in test_tar_filter    filtered = tarfile.tar_filter(tarinfo, '')  File "/home/serhiy/py/cpython/Lib/tarfile.py", line 836, in tar_filter    new_attrs = _get_filtered_attrs(member, dest_path, False)  File "/home/serhiy/py/cpython/Lib/tarfile.py", line 782, in _get_filtered_attrs    target_path = os.path.realpath(os.path.join(dest_path, name))  File "/home/serhiy/py/cpython/Lib/posixpath.py", line 405, in realpath    return _realpath(filename, strict, sep, curdir, pardir, getcwd)  File "/home/serhiy/py/cpython/Lib/posixpath.py", line 452, in _realpath    st_mode = lstat(newpath).st_mode              ~~~~~^^^^^^^^^  File "/home/serhiy/py/cpython/Lib/encodings/koi8_u.py", line 12, in encode    return codecs.charmap_encode(input,errors,encoding_table)           ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^UnicodeEncodeError: 'charmap' codec can't encode characters in position 69-75: character maps to <undefined>encoding with 'koi8-u' codec failed----------------------------------------------------------------------

This happens because they useos.path.realpath() for paths in a tar archive, which usesos.stat(), which fails with unexpectedUnicodeEncodeError if the path in a tar archive can't be encoded in the current filesystem encoding. This error should be handled at some level, either inos.path.realpath() or intarfile.os.stat() can also raiseValueError if the path contain null bytes. Don't know if this is relevant here, we should test.

Linked PRs

Metadata

Metadata

Labels

3.12only security fixes3.13bugs and security fixes3.14bugs and security fixesstdlibPython modules in the Lib dirtype-bugAn unexpected behavior, bug, or error

Projects

Status

Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions


    [8]ページ先頭

    ©2009-2025 Movatter.jp