This PEP discusses supporting access to all files possible onWindows NT by passing Unicode file names directly to the system’swide-character functions.
Python 2.2 on Win32 platforms converts Unicode file names passedto open and to functions in theos module into the ‘mbcs’ encodingbefore passing the result to the operating system. This is oftensuccessful in the common case where the script is operating withthe locale set to the same value as when the file was created.Most machines are set up as one locale and rarely if ever changedfrom this locale. For some users, locale is changed more oftenand on servers there are often files saved by users usingdifferent locales.
On Windows NT and descendent operating systems, including Windows2000 and Windows XP, wide-character APIs are available thatprovide direct access to all file names, including those that arenot representable using the current locale. The purpose of thisproposal is to provide access to these wide-character APIs throughthe standard Python file object and posix module and so provideaccess to all files on Windows NT.
On Windows platforms which provide wide-character file APIs, whenUnicode arguments are provided to file APIs, wide-character callsare made instead of the standard C library and posix calls.
The Python file object is extended to use a Unicode file nameargument directly rather than converting it. This affects thefile object constructorfile(filename[,mode[,bufsize]]) and alsotheopen function which is an alias of this constructor. When aUnicode filename argument is used here then thename attribute ofthe file object will be Unicode. The representation of a fileobject,repr(f) will display Unicode file names as an escapedstring in a similar manner to the representation of Unicodestrings.
Theposix module contains functions that take file or directorynames:chdir,listdir,mkdir,open,remove,rename,rmdir,stat, and_getfullpathname. These will use Unicodearguments directly rather than converting them. For therename function, thisbehaviour is triggered when either of the arguments is Unicode andthe other argument converted to Unicode using the defaultencoding.
Thelistdir function currently returns a list of strings. Underthis proposal, it will return a list of Unicode strings when itspath argument is Unicode.
On the consumer Windows operating systems, Windows 95, Windows 98,and Windows ME, there are no wide-character file APIs so behaviouris unchanged under this proposal. It may be possible in thefuture to extend this proposal to cover these operating systems asthe VFAT-32 file system used by them does support Unicode filenames but access is difficult and so implementing this wouldrequire much work. The “Microsoft Layer for Unicode” could be astarting point for implementing this.
Python can be compiled with the size of Unicode characters set to4 bytes rather than 2 by definingPY_UNICODE_TYPE to be a 4 bytetype andPy_UNICODE_SIZE to be 4. As the Windows API does notaccept 4 byte characters, the features described in this proposalwill not work in this mode so the implementation falls back to thecurrent ‘mbcs’ encoding technique. This restriction could be liftedin the future by performing extra conversions usingPyUnicode_AsWideChar but for now that would add too muchcomplexity for a very rarely used feature.
The implementation is available at[2].
[1] Microsoft Windows APIshttps://msdn.microsoft.com/
This document has been placed in the public domain.
Source:https://github.com/python/peps/blob/main/peps/pep-0277.rst
Last modified:2025-02-01 08:55:40 GMT