Movatterモバイル変換


[0]ホーム

URL:


Following system colour schemeSelected dark colour schemeSelected light colour scheme

Python Enhancement Proposals

PEP 587 – Python Initialization Configuration

Author:
Victor Stinner <vstinner at python.org>, Alyssa Coghlan <ncoghlan at gmail.com>
BDFL-Delegate:
Thomas Wouters <thomas at python.org>
Discussions-To:
Python-Dev list
Status:
Final
Type:
Standards Track
Created:
27-Mar-2019
Python-Version:
3.8

Table of Contents

Abstract

Add a new C API to configure the Python Initialization providing finercontrol on the whole configuration and better error reporting.

It becomes possible to read the configuration and then override somecomputed parameters before it is applied. It also becomes possible tocompletely override how Python computes the module search paths(sys.path).

The newIsolated Configuration provides sane default values toisolate Python from the system. For example, to embed Python into anapplication. Using the environment are now opt-in options, rather thanan opt-out options. For example, environment variables, command linearguments and global configuration variables are ignored by default.

Building a customized Python which behaves as the regular Python becomeseasier using the newPy_RunMain() function. Moreover, using thePython Configuration,PyConfig.argv arguments are now parsed thesame way the regular Python parses command line arguments, andPyConfig.xoptions are handled as-Xopt command line options.

This extracts a subset of the API design from thePEP 432 developmentand refactoring work that is now considered sufficiently stable to makepublic (allowing 3rd party embedding applications access to the sameconfiguration APIs that the native CPython CLI is now using).

Rationale

Python is highly configurable but its configuration evolved organically.The initialization configuration is scattered all around the code usingdifferent ways to set them: global configuration variables (ex:Py_IsolatedFlag), environment variables (ex:PYTHONPATH),command line arguments (ex:-b), configuration files (ex:pyvenv.cfg), function calls (ex:Py_SetProgramName()). Astraightforward and reliable way to configure Python is needed.

Some configuration parameters are not accessible from the C API, or noteasily. For example, there is no API to override the default values ofsys.executable.

Some options likePYTHONPATH can only be set using an environmentvariable which has a side effect on Python child processes if not unsetproperly.

Some options also depends on other options: seePriority and Rules.Python 3.7 API does not provide a consistent view of the overallconfiguration.

The C API of Python 3.7 Initialization takeswchar_t* strings asinput whereas the Python filesystem encoding is set during theinitialization which can lead to mojibake.

Python 3.7 APIs likePy_Initialize() aborts the process on memoryallocation failure which is not convenient when Python is embedded.Moreover,Py_Main() could exit directly the process rather thanreturning an exit code. Proposed new API reports the error or exit codeto the caller which can decide how to handle it.

Implementing thePEP 540 (UTF-8 Mode) and the new-Xdev correctlywas almost impossible in Python 3.6. The code base has been deeplyreworked in Python 3.7 and then in Python 3.8 to read the configurationinto a structure with no side effect. It becomes possible to clear theconfiguration (release memory) and read again the configuration if theencoding changed . It is required to implement properly the UTF-8 whichchanges the encoding using-Xutf8 command line option. Internally,bytesargv strings are decoded from the filesystem encoding. The-Xdev changes the memory allocator (behaves asPYTHONMALLOC=debug), whereas it was not possible to change thememory allocationwhile parsing the command line arguments. The newdesign of the internal implementation not only allowed to implementproperly-Xutf8 and-Xdev, it also allows to change the Pythonbehavior way more easily, especially for corner cases like that, andensure that the configuration remains consistent: seePriority andRules.

This PEP is a partial implementation ofPEP 432 which is the overalldesign. New fields can be added later toPyConfig structure tofinish the implementation of thePEP 432 (e.g. by adding a new partialinitialization API which allows to configure Python using Python objects tofinish the full initialization). However, those features are omitted from thisPEP as even the native CPython CLI doesn’t work that way - the public APIproposal in this PEP is limited to features which have already been implementedand adopted as private APIs for us in the native CPython CLI.

Python Initialization C API

This PEP proposes to add the following new structures and functions.

New structures:

  • PyConfig
  • PyPreConfig
  • PyStatus
  • PyWideStringList

New functions:

  • PyConfig_Clear(config)
  • PyConfig_InitIsolatedConfig()
  • PyConfig_InitPythonConfig()
  • PyConfig_Read(config)
  • PyConfig_SetArgv(config,argc,argv)
  • PyConfig_SetBytesArgv(config,argc,argv)
  • PyConfig_SetBytesString(config,config_str,str)
  • PyConfig_SetString(config,config_str,str)
  • PyPreConfig_InitIsolatedConfig(preconfig)
  • PyPreConfig_InitPythonConfig(preconfig)
  • PyStatus_Error(err_msg)
  • PyStatus_Exception(status)
  • PyStatus_Exit(exitcode)
  • PyStatus_IsError(status)
  • PyStatus_IsExit(status)
  • PyStatus_NoMemory()
  • PyStatus_Ok()
  • PyWideStringList_Append(list,item)
  • PyWideStringList_Insert(list,index,item)
  • Py_BytesMain(argc,argv)
  • Py_ExitStatusException(status)
  • Py_InitializeFromConfig(config)
  • Py_PreInitialize(preconfig)
  • Py_PreInitializeFromArgs(preconfig,argc,argv)
  • Py_PreInitializeFromBytesArgs(preconfig,argc,argv)
  • Py_RunMain()

This PEP also adds_PyRuntimeState.preconfig (PyPreConfig type)andPyInterpreterState.config (PyConfig type) fields to theseinternal structures.PyInterpreterState.config becomes the newreference configuration, replacing global configuration variables andother private variables.

PyWideStringList

PyWideStringList is a list ofwchar_t* strings.

PyWideStringList structure fields:

  • length (Py_ssize_t)
  • items (wchar_t**)

Methods:

  • PyStatusPyWideStringList_Append(PyWideStringList*list,constwchar_t*item):Appenditem tolist.
  • PyStatusPyWideStringList_Insert(PyWideStringList*list,Py_ssize_tindex,constwchar_t*item):Insertitem intolist atindex. Ifindex is greater thanlist length, just appenditem tolist.

Iflength is non-zero,items must be non-NULL and all strings mustbe non-NULL.

PyStatus

PyStatus is a structure to store the status of an initializationfunction: success, error or exit. For an error, it can store the Cfunction name which created the error.

Example:

PyStatusalloc(void**ptr,size_tsize){*ptr=PyMem_RawMalloc(size);if(*ptr==NULL){returnPyStatus_NoMemory();}returnPyStatus_Ok();}intmain(intargc,char**argv){void*ptr;PyStatusstatus=alloc(&ptr,16);if(PyStatus_Exception(status)){Py_ExitStatusException(status);}PyMem_Free(ptr);return0;}

PyStatus fields:

  • exitcode (int):Argument passed toexit().
  • err_msg (constchar*):Error message.
  • func (constchar*):Name of the function which created an error, can beNULL.
  • private_type field: for internal usage only.

Functions to create a status:

  • PyStatus_Ok(): Success.
  • PyStatus_Error(err_msg): Initialization error with a message.
  • PyStatus_NoMemory(): Memory allocation failure (out of memory).
  • PyStatus_Exit(exitcode): Exit Python with the specified exit code.

Functions to handle a status:

  • PyStatus_Exception(status): Is the result an error or an exit?If true, the exception must be handled; by callingPy_ExitStatusException(status) for example.
  • PyStatus_IsError(status): Is the result an error?
  • PyStatus_IsExit(status): Is the result an exit?
  • Py_ExitStatusException(status): Callexit(exitcode) ifstatusis an exit. Print the error messageand exit with a non-zero exit codeifstatus is an error. Must only be called ifPyStatus_Exception(status) is true.

Note

Internally, Python uses macros which setPyStatus.func,whereas functions to create a status setfunc toNULL.

Preinitialization with PyPreConfig

ThePyPreConfig structure is used to preinitialize Python:

  • Set the Python memory allocator
  • Configure the LC_CTYPE locale
  • Set the UTF-8 mode

Example using the preinitialization to enable the UTF-8 Mode:

PyStatusstatus;PyPreConfigpreconfig;PyPreConfig_InitPythonConfig(&preconfig);preconfig.utf8_mode=1;status=Py_PreInitialize(&preconfig);if(PyStatus_Exception(status)){Py_ExitStatusException(status);}/*atthispoint,PythonwillspeakUTF-8*/Py_Initialize();/*...usePythonAPIhere...*/Py_Finalize();

Function to initialize a preconfiguration:

  • PyStatusPyPreConfig_InitIsolatedConfig(PyPreConfig*preconfig)
  • PyStatusPyPreConfig_InitPythonConfig(PyPreConfig*preconfig)

Functions to preinitialize Python:

  • PyStatusPy_PreInitialize(constPyPreConfig*preconfig)
  • PyStatusPy_PreInitializeFromBytesArgs(constPyPreConfig*preconfig,intargc,char*const*argv)
  • PyStatusPy_PreInitializeFromArgs(constPyPreConfig*preconfig,intargc,wchar_t*const*argv)

The caller is responsible to handle exceptions (error or exit) usingPyStatus_Exception() andPy_ExitStatusException().

ForPython Configuration (PyPreConfig_InitPythonConfig()),if Python is initialized with command line arguments, the command linearguments must also be passed to preinitialize Python, since they havean effect on the pre-configuration like encodings. For example, the-Xutf8 command line option enables the UTF-8 Mode.

PyPreConfig fields:

  • allocator (int):Name of the memory allocator (ex:PYMEM_ALLOCATOR_MALLOC). Validvalues:
    • PYMEM_ALLOCATOR_NOT_SET (0): don’t change memory allocators(use defaults)
    • PYMEM_ALLOCATOR_DEFAULT (1): default memory allocators
    • PYMEM_ALLOCATOR_DEBUG (2): default memory allocators withdebug hooks
    • PYMEM_ALLOCATOR_MALLOC (3): force usage ofmalloc()
    • PYMEM_ALLOCATOR_MALLOC_DEBUG (4): force usage ofmalloc() with debug hooks
    • PYMEM_ALLOCATOR_PYMALLOC (5): Python “pymalloc” allocator
    • PYMEM_ALLOCATOR_PYMALLOC_DEBUG (6): Python “pymalloc”allocator with debug hooks
    • Note:PYMEM_ALLOCATOR_PYMALLOC andPYMEM_ALLOCATOR_PYMALLOC_DEBUG are not supported if Python isconfigured using--without-pymalloc
  • configure_locale (int):Set the LC_CTYPE locale to the user preferred locale? If equals to 0,setcoerce_c_locale andcoerce_c_locale_warn to 0.
  • coerce_c_locale (int):If equals to 2, coerce the C locale; if equals to 1, read the LC_CTYPElocale to decide if it should be coerced.
  • coerce_c_locale_warn (int):If non-zero, emit a warning if the C locale is coerced.
  • dev_mode (int):SeePyConfig.dev_mode.
  • isolated (int):SeePyConfig.isolated.
  • legacy_windows_fs_encoding (int, Windows only):If non-zero, disable UTF-8 Mode, set the Python filesystem encoding tombcs, set the filesystem error handler toreplace.
  • parse_argv (int):If non-zero,Py_PreInitializeFromArgs() andPy_PreInitializeFromBytesArgs() parse theirargv argument thesame way the regular Python parses command line arguments: seeCommand Line Arguments.
  • use_environment (int):SeePyConfig.use_environment.
  • utf8_mode (int):If non-zero, enable the UTF-8 mode.

Thelegacy_windows_fs_encoding field is only available on Windows.#ifdefMS_WINDOWS macro can be used for Windows specific code.

PyPreConfig private fields, for internal use only:

  • _config_init (int):Function used to initializePyConfig, used for preinitialization.

PyMem_SetAllocator() can be called afterPy_PreInitialize() andbeforePy_InitializeFromConfig() to install a custom memoryallocator. It can be called beforePy_PreInitialize() ifallocator is set toPYMEM_ALLOCATOR_NOT_SET (default value).

Python memory allocation functions likePyMem_RawMalloc() must notbe used before Python preinitialization, whereas calling directlymalloc() andfree() is always safe.Py_DecodeLocale() mustnot be called before the preinitialization.

Initialization with PyConfig

ThePyConfig structure contains most parameters to configure Python.

Example setting the program name:

voidinit_python(void){PyStatusstatus;PyConfigconfig;PyConfig_InitPythonConfig(&config);/*Settheprogramname.ImplicitlypreinitializePython.*/status=PyConfig_SetString(&config,&config.program_name,L"/path/to/my_program");if(PyStatus_Exception(status)){gotofail;}status=Py_InitializeFromConfig(&config);if(PyStatus_Exception(status)){gotofail;}PyConfig_Clear(&config);return;fail:PyConfig_Clear(&config);Py_ExitStatusException(status);}

PyConfig methods:

  • voidPyConfig_InitPythonConfig(PyConfig*config)Initialize configuration withPython Configuration.
  • voidPyConfig_InitIsolatedConfig(PyConfig*config):Initialize configuration withIsolated Configuration.
  • PyStatusPyConfig_SetString(PyConfig*config,wchar_t*const*config_str,constwchar_t*str):Copy the wide character stringstr into*config_str.Preinitialize Python if needed.
  • PyStatusPyConfig_SetBytesString(PyConfig*config,wchar_t*const*config_str,constchar*str):Decodestr usingPy_DecodeLocale() and set the result into*config_str.Preinitialize Python if needed.
  • PyStatusPyConfig_SetArgv(PyConfig*config,intargc,wchar_t*const*argv):Set command line arguments from wide character strings.Preinitialize Python if needed.
  • PyStatusPyConfig_SetBytesArgv(PyConfig*config,intargc,char*const*argv):Set command line arguments: decode bytes usingPy_DecodeLocale().Preinitialize Python if needed.
  • PyStatusPyConfig_Read(PyConfig*config):Read all Python configuration. Fields which are already initializedare left unchanged.Preinitialize Python if needed.
  • voidPyConfig_Clear(PyConfig*config):Release configuration memory.

MostPyConfig methods preinitialize Python if needed. In that case,the Python preinitialization configuration in based on thePyConfig.If configuration fields which are in common withPyPreConfig aretuned, they must be set before calling aPyConfig method:

  • dev_mode
  • isolated
  • parse_argv
  • use_environment

Moreover, ifPyConfig_SetArgv() orPyConfig_SetBytesArgv() isused, this method must be called first, before other methods, since thepreinitialization configuration depends on command line arguments (ifparse_argv is non-zero).

Functions to initialize Python:

  • PyStatusPy_InitializeFromConfig(constPyConfig*config):Initialize Python fromconfig configuration.

The caller of these methods and functions is responsible to handleexceptions (error or exit) usingPyStatus_Exception() andPy_ExitStatusException().

PyConfig fields:

  • argv (PyWideStringList):Command line arguments,sys.argv. Seeparse_argv to parseargv the same way the regular Python parses Python command linearguments. Ifargv is empty, an empty string is added to ensurethatsys.argv always exists and is never empty.
  • base_exec_prefix (wchar_t*):sys.base_exec_prefix.
  • base_prefix (wchar_t*):sys.base_prefix.
  • buffered_stdio (int):If equals to 0, enable unbuffered mode, making the stdout and stderr streamsunbuffered.
  • bytes_warning (int):If equals to 1, issue a warning when comparingbytes orbytearray withstr, or comparingbytes withint. Ifequal or greater to 2, raise aBytesWarning exception.
  • check_hash_pycs_mode (wchar_t*):--check-hash-based-pycs command line option value (seePEP 552).Valid values:always,never anddefault. The default valueisdefault.
  • configure_c_stdio (int):If non-zero, configure C standard streams (stdio,stdout,stdout). For example, set their mode toO_BINARY on Windows.
  • dev_mode (int):Development mode
  • dump_refs (int):If non-zero, dump all objects which are still alive at exit.Require a special Python build withPy_REF_DEBUG macro defined.
  • exec_prefix (wchar_t*):sys.exec_prefix.
  • executable (wchar_t*):sys.executable.
  • faulthandler (int):If non-zero, callfaulthandler.enable().
  • filesystem_encoding (wchar_t*):Filesystem encoding,sys.getfilesystemencoding().
  • filesystem_errors (wchar_t*):Filesystem encoding errors,sys.getfilesystemencodeerrors().
  • use_hash_seed (int),hash_seed (unsignedlong):Randomized hash function seed.
  • home (wchar_t*):Python home directory.
  • import_time (int):If non-zero, profile import time.
  • inspect (int):Enter interactive mode after executing a script or a command.
  • install_signal_handlers (int):Install signal handlers?
  • interactive (int):Interactive mode.
  • isolated (int):If greater than 0, enable isolated mode:
    • sys.path contains neither the script’s directory (computed fromargv[0] or the current directory) nor the user’s site-packagesdirectory.
    • Python REPL doesn’t importreadline nor enable default readlineconfiguration on interactive prompts.
    • Setuse_environment anduser_site_directory to 0.
  • legacy_windows_stdio (int, Windows only):If non-zero, useio.FileIO instead ofWindowsConsoleIO forsys.stdin,sys.stdout andsys.stderr.
  • malloc_stats (int):If non-zero, dump statistics onpymalloc memory allocator at exit.The option is ignored if Python is built using--without-pymalloc.
  • pythonpath_env (wchar_t*):Module search paths as a string separated by DELIM (usually:character).Initialized fromPYTHONPATH environment variable value by default.
  • module_search_paths_set (int),module_search_paths (PyWideStringList):sys.path. Ifmodule_search_paths_set is equal to 0, themodule_search_paths is overridden by the function computing thePath Configuration.
  • optimization_level (int):Compilation optimization level:
    • 0: Peephole optimizer (and__debug__ is set toTrue)
    • 1: Remove assertions, set__debug__ toFalse
    • 2: Strip docstrings
  • parse_argv (int):If non-zero, parseargv the same way the regular Python commandline arguments, and strip Python arguments fromargv: seeCommandLine Arguments.
  • parser_debug (int):If non-zero, turn on parser debugging output (for expert only,depending on compilation options).
  • pathconfig_warnings (int):If equal to 0, suppress warnings when computing the pathconfiguration (Unix only, Windows does not log any warning). Otherwise,warnings are written into stderr.
  • prefix (wchar_t*):sys.prefix.
  • program_name (wchar_t*):Program name.
  • pycache_prefix (wchar_t*):.pyc cache prefix.
  • quiet (int):Quiet mode. For example, don’t display the copyright and versionmessages even in interactive mode.
  • run_command (wchar_t*):python3-cCOMMAND argument.
  • run_filename (wchar_t*):python3FILENAME argument.
  • run_module (wchar_t*):python3-mMODULE argument.
  • show_alloc_count (int):Show allocation counts at exit?Need a special Python build withCOUNT_ALLOCS macro defined.
  • show_ref_count (int):Show total reference count at exit?Need a debug build of Python (Py_REF_DEBUG macro should bedefined).
  • site_import (int):Import thesite module at startup?
  • skip_source_first_line (int):Skip the first line of the source?
  • stdio_encoding (wchar_t*),stdio_errors (wchar_t*):Encoding and encoding errors ofsys.stdin,sys.stdoutandsys.stderr.
  • tracemalloc (int):If non-zero, calltracemalloc.start(value).
  • user_site_directory (int):If non-zero, add user site directory tosys.path.
  • verbose (int):If non-zero, enable verbose mode.
  • warnoptions (PyWideStringList):Options of thewarnings module to build warnings filters.
  • write_bytecode (int):If non-zero, write.pyc files.
  • xoptions (PyWideStringList):sys._xoptions.

Thelegacy_windows_stdio field is only available on Windows.#ifdefMS_WINDOWS macro can be used for Windows specific code.

Ifparse_argv is non-zero,argv arguments are parsed the sameway the regular Python parses command line arguments, and Pythonarguments are stripped fromargv: seeCommand Line Arguments.

Thexoptions options are parsed to set other options: see-XOptions.

PyConfig private fields, for internal use only:

  • _config_init (int):Function used to initializePyConfig, used for preinitialization.
  • _install_importlib (int):Install importlib?
  • _init_main (int):If equal to 0, stop Python initialization before the “main” phase(seePEP 432).

More complete example modifying the default configuration, read theconfiguration, and then override some parameters:

PyStatusinit_python(constchar*program_name){PyStatusstatus;PyConfigconfig;PyConfig_InitPythonConfig(&config);/*Settheprogramnamebeforereadingtheconfiguration(decodebytestringfromthelocaleencoding).ImplicitlypreinitializePython.*/status=PyConfig_SetBytesString(&config,&config.program_name,program_name);if(PyStatus_Exception(status)){gotodone;}/*Readallconfigurationatonce*/status=PyConfig_Read(&config);if(PyStatus_Exception(status)){gotodone;}/*Appendourcustomsearchpathtosys.path*/status=PyWideStringList_Append(&config.module_search_paths,L"/path/to/more/modules");if(PyStatus_Exception(status)){gotodone;}/*OverrideexecutablecomputedbyPyConfig_Read()*/status=PyConfig_SetString(&config,&config.executable,L"/path/to/my_executable");if(PyStatus_Exception(status)){gotodone;}status=Py_InitializeFromConfig(&config);done:PyConfig_Clear(&config);returnstatus;}

Note

PyImport_FrozenModules,PyImport_AppendInittab() andPyImport_ExtendInittab() functions are still relevant andcontinue to work as previously. They should be set or called afterPython preinitialization and before the Python initialization.

Isolated Configuration

PyPreConfig_InitIsolatedConfig() andPyConfig_InitIsolatedConfig() functions create a configuration toisolate Python from the system. For example, to embed Python into anapplication.

This configuration ignores global configuration variables, environmentsvariables and command line arguments (argv is not parsed). The Cstandard streams (ex:stdout) and the LC_CTYPE locale are leftunchanged by default.

Configuration files are still used with this configuration. Set thePath Configuration (“output fields”) to ignore these configurationfiles and avoid the function computing the default path configuration.

Python Configuration

PyPreConfig_InitPythonConfig() andPyConfig_InitPythonConfig()functions create a configuration to build a customized Python whichbehaves as the regular Python.

Environments variables and command line arguments are used to configurePython, whereas global configuration variables are ignored.

This function enables C locale coercion (PEP 538) and UTF-8 Mode (PEP540) depending on the LC_CTYPE locale,PYTHONUTF8 andPYTHONCOERCECLOCALE environment variables.

Example of customized Python always running in isolated mode:

intmain(intargc,char**argv){PyStatusstatus;PyConfigconfig;PyConfig_InitPythonConfig(&config);config.isolated=1;/*Decodecommandlinearguments.ImplicitlypreinitializePython(inisolatedmode).*/status=PyConfig_SetBytesArgv(&config,argc,argv);if(PyStatus_Exception(status)){gotofail;}status=Py_InitializeFromConfig(&config);if(PyStatus_Exception(status)){gotofail;}PyConfig_Clear(&config);returnPy_RunMain();fail:PyConfig_Clear(&config);if(PyStatus_IsExit(status)){returnstatus.exitcode;}/*Displaytheerrormessageandexittheprocesswithnon-zeroexitcode*/Py_ExitStatusException(status);}

This example is a basic implementation of the “System Python Executable”discussed inPEP 432.

Path Configuration

PyConfig contains multiple fields for the path configuration:

  • Path configuration input fields:
    • home
    • pythonpath_env
    • pathconfig_warnings
  • Path configuration output fields:
    • exec_prefix
    • executable
    • prefix
    • module_search_paths_set,module_search_paths

If at least one “output field” is not set, Python computes the pathconfiguration to fill unset fields. Ifmodule_search_paths_set isequal to 0,module_search_paths is overridden andmodule_search_paths_set is set to 1.

It is possible to completely ignore the function computing the defaultpath configuration by setting explicitly all path configuration outputfields listed above. A string is considered as set even if it is non-empty.module_search_paths is considered as set ifmodule_search_paths_set is set to 1. In this case, pathconfiguration input fields are ignored as well.

Setpathconfig_warnings to 0 to suppress warnings when computing thepath configuration (Unix only, Windows does not log any warning).

Ifbase_prefix orbase_exec_prefix fields are not set, theyinherit their value fromprefix andexec_prefix respectively.

Py_RunMain() andPy_Main() modifysys.path:

  • Ifrun_filename is set and is a directory which contains a__main__.py script, prependrun_filename tosys.path.
  • Ifisolated is zero:
    • Ifrun_module is set, prepend the current directory tosys.path. Do nothing if the current directory cannot be read.
    • Ifrun_filename is set, prepends the directory of the filenametosys.path.
    • Otherwise, prepends an empty string tosys.path.

Ifsite_import is non-zero,sys.path can be modified by thesite module. Ifuser_site_directory is non-zero and the user’ssite-package directory exists, thesite module appends the user’ssite-package directory tosys.path.

See alsoConfiguration Files used by the path configuration.

Py_BytesMain()

Python 3.7 provides a high-levelPy_Main() function which requiresto pass command line arguments aswchar_t* strings. It isnon-trivial to use the correct encoding to decode bytes. Python has itsown set of issues with C locale coercion and UTF-8 Mode.

This PEP adds a newPy_BytesMain() function which takes command linearguments as bytes:

intPy_BytesMain(intargc,char**argv)

Py_RunMain()

The newPy_RunMain() function executes the command(PyConfig.run_command), the script (PyConfig.run_filename) orthe module (PyConfig.run_module) specified on the command line or inthe configuration, and then finalizes Python. It returns an exit statusthat can be passed to theexit() function.

intPy_RunMain(void);

SeePython Configuration for an example of customized Python alwaysrunning in isolated mode usingPy_RunMain().

Multi-Phase Initialization Private Provisional API

This section is a private provisional API introducing multi-phaseinitialization, the core feature of thePEP 432:

  • “Core” initialization phase, “bare minimum Python”:
    • Builtin types;
    • Builtin exceptions;
    • Builtin and frozen modules;
    • Thesys module is only partially initialized(ex:sys.path doesn’t exist yet);
  • “Main” initialization phase, Python is fully initialized:
    • Install and configureimportlib;
    • Apply thePath Configuration;
    • Install signal handlers;
    • Finishsys module initialization (ex: createsys.stdout andsys.path);
    • Enable optional features likefaulthandler andtracemalloc;
    • Import thesite module;
    • etc.

Private provisional API:

  • PyConfig._init_main: if set to 0,Py_InitializeFromConfig()stops at the “Core” initialization phase.
  • PyStatus_Py_InitializeMain(void): move to the “Main”initialization phase, finish the Python initialization.

No module is imported during the “Core” phase and theimportlibmodule is not configured: thePath Configuration is only appliedduring the “Main” phase. It may allow to customize Python in Python tooverride or tune thePath Configuration, maybe install a customsys.meta_path importer or an import hook, etc.

It may become possible to compute thePath Configuration in Python,after the Core phase and before the Main phase, which is one of the PEP432 motivation.

The “Core” phase is not properly defined: what should be and what shouldnot be available at this phase is not specified yet. The API is markedas private and provisional: the API can be modified or even be removedanytime until a proper public API is designed.

Example running Python code between “Core” and “Main” initializationphases:

voidinit_python(void){PyStatusstatus;PyConfigconfig;PyConfig_InitPythonConfig(&config);config._init_main=0;/*...customize'config'configuration...*/status=Py_InitializeFromConfig(&config);PyConfig_Clear(&config);if(PyStatus_Exception(status)){Py_ExitStatusException(status);}/*Usesys.stderrbecausesys.stdoutisonlycreatedby_Py_InitializeMain()*/intres=PyRun_SimpleString("import sys; ""print('Run Python code before _Py_InitializeMain', ""file=sys.stderr)");if(res<0){exit(1);}/*...putmoreconfigurationcodehere...*/status=_Py_InitializeMain();if(PyStatus_Exception(status)){Py_ExitStatusException(status);}}

Backwards Compatibility

This PEP only adds a new API: it leaves the existing API unchanged andhas no impact on the backwards compatibility.

The Python 3.7Py_Initialize() function now disable the C localecoercion (PEP 538) and the UTF-8 Mode (PEP 540) by default to preventmojibake. The new API using thePython Configuration is needed toenable them automatically.

Annexes

Comparison of Python and Isolated Configurations

Differences betweenPyPreConfig_InitPythonConfig()andPyPreConfig_InitIsolatedConfig():

PyPreConfigPythonIsolated
coerce_c_locale_warn-10
coerce_c_locale-10
configure_locale10
dev_mode-10
isolated01
legacy_windows_fs_encoding-10
use_environment00
parse_argv10
utf8_mode-10

Differences betweenPyConfig_InitPythonConfig()andPyConfig_InitIsolatedConfig():

PyConfigPythonIsolated
configure_c_stdio10
install_signal_handlers10
isolated01
parse_argv10
pathconfig_warnings10
use_environment10
user_site_directory10

Priority and Rules

Priority of configuration parameters, highest to lowest:

  • PyConfig
  • PyPreConfig
  • Configuration files
  • Command line options
  • Environment variables
  • Global configuration variables

Priority of warning options, highest to lowest:

  • PyConfig.warnoptions
  • PySys_AddWarnOption()
  • PyConfig.bytes_warning (add"error::BytesWarning" filter ifgreater than 1, add"default::BytesWarning filter if equals to 1)
  • -Wopt command line argument
  • PYTHONWARNINGS environment variable
  • PyConfig.dev_mode (add"default" filter)

Rules onPyConfig parameters:

  • Ifisolated is non-zero,use_environment anduser_site_directory are set to 0.
  • Ifdev_mode is non-zero,allocator is set to"debug",faulthandler is set to 1, and"default" filter is added towarnoptions. But thePYTHONMALLOC environment variable has thepriority overdev_mode to set the memory allocator.
  • Ifbase_prefix is not set, it inheritsprefix value.
  • Ifbase_exec_prefix is not set, it inheritsexec_prefix value.
  • If thepython._pth configuration file is present,isolated isset to 1 andsite_import is set to 0; butsite_import is setto 1 ifpython._pth containsimportsite.

Rules onPyConfig andPyPreConfig parameters:

  • IfPyPreConfig.legacy_windows_fs_encoding is non-zero,setPyPreConfig.utf8_mode to 0, setPyConfig.filesystem_encoding tombcs, and setPyConfig.filesystem_errors toreplace.

Configuration Files

Python configuration files used by thePath Configuration:

  • pyvenv.cfg
  • python._pth (Windows only)
  • pybuilddir.txt (Unix only)

Global Configuration Variables

Global configuration variables mapped toPyPreConfig fields:

VariableField
Py_IgnoreEnvironmentFlaguse_environment (NOT)
Py_IsolatedFlagisolated
Py_LegacyWindowsFSEncodingFlaglegacy_windows_fs_encoding
Py_UTF8Modeutf8_mode

(NOT) means that thePyPreConfig value is the opposite of the globalconfiguration variable value.Py_LegacyWindowsFSEncodingFlag is onlyavailable on Windows.

Global configuration variables mapped toPyConfig fields:

VariableField
Py_BytesWarningFlagbytes_warning
Py_DebugFlagparser_debug
Py_DontWriteBytecodeFlagwrite_bytecode (NOT)
Py_FileSystemDefaultEncodeErrorsfilesystem_errors
Py_FileSystemDefaultEncodingfilesystem_encoding
Py_FrozenFlagpathconfig_warnings (NOT)
Py_HasFileSystemDefaultEncodingfilesystem_encoding
Py_HashRandomizationFlaguse_hash_seed,hash_seed
Py_IgnoreEnvironmentFlaguse_environment (NOT)
Py_InspectFlaginspect
Py_InteractiveFlaginteractive
Py_IsolatedFlagisolated
Py_LegacyWindowsStdioFlaglegacy_windows_stdio
Py_NoSiteFlagsite_import (NOT)
Py_NoUserSiteDirectoryuser_site_directory (NOT)
Py_OptimizeFlagoptimization_level
Py_QuietFlagquiet
Py_UnbufferedStdioFlagbuffered_stdio (NOT)
Py_VerboseFlagverbose
_Py_HasFileSystemDefaultEncodeErrorsfilesystem_errors

(NOT) means that thePyConfig value is the opposite of the globalconfiguration variable value.Py_LegacyWindowsStdioFlag is onlyavailable on Windows.

Command Line Arguments

Usage:

python3[options]python3[options]-cCOMMANDpython3[options]-mMODULEpython3[options]SCRIPT

Command line options mapped to pseudo-action onPyPreConfig fields:

OptionPyConfig field
-Euse_environment=0
-Iisolated=1
-Xdevdev_mode=1
-Xutf8utf8_mode=1
-Xutf8=VALUEutf8_mode=VALUE

Command line options mapped to pseudo-action onPyConfig fields:

OptionPyConfig field
-bbytes_warning++
-Bwrite_bytecode=0
-cCOMMANDrun_command=COMMAND
--check-hash-based-pycs=MODEcheck_hash_pycs_mode=MODE
-dparser_debug++
-Euse_environment=0
-iinspect++ andinteractive++
-Iisolated=1
-mMODULErun_module=MODULE
-Ooptimization_level++
-qquiet++
-Ruse_hash_seed=0
-suser_site_directory=0
-Ssite_import
-tignored (kept for backwards compatibility)
-ubuffered_stdio=0
-vverbose++
-WWARNINGaddWARNING towarnoptions
-xskip_source_first_line=1
-XOPTIONaddOPTION toxoptions

-h,-? and-V options are handled withoutPyConfig.

-X Options

-X options mapped to pseudo-action onPyConfig fields:

OptionPyConfig field
-Xdevdev_mode=1
-Xfaulthandlerfaulthandler=1
-Ximporttimeimport_time=1
-Xpycache_prefix=PREFIXpycache_prefix=PREFIX
-Xshowalloccountshow_alloc_count=1
-Xshowrefcountshow_ref_count=1
-Xtracemalloc=Ntracemalloc=N

Environment Variables

Environment variables mapped toPyPreConfig fields:

VariablePyPreConfig field
PYTHONCOERCECLOCALEcoerce_c_locale,coerce_c_locale_warn
PYTHONDEVMODEdev_mode
PYTHONLEGACYWINDOWSFSENCODINGlegacy_windows_fs_encoding
PYTHONMALLOCallocator
PYTHONUTF8utf8_mode

Environment variables mapped toPyConfig fields:

VariablePyConfig field
PYTHONDEBUGparser_debug
PYTHONDEVMODEdev_mode
PYTHONDONTWRITEBYTECODEwrite_bytecode
PYTHONDUMPREFSdump_refs
PYTHONEXECUTABLEprogram_name
PYTHONFAULTHANDLERfaulthandler
PYTHONHASHSEEDuse_hash_seed,hash_seed
PYTHONHOMEhome
PYTHONINSPECTinspect
PYTHONIOENCODINGstdio_encoding,stdio_errors
PYTHONLEGACYWINDOWSSTDIOlegacy_windows_stdio
PYTHONMALLOCSTATSmalloc_stats
PYTHONNOUSERSITEuser_site_directory
PYTHONOPTIMIZEoptimization_level
PYTHONPATHpythonpath_env
PYTHONPROFILEIMPORTTIMEimport_time
PYTHONPYCACHEPREFIX,pycache_prefix
PYTHONTRACEMALLOCtracemalloc
PYTHONUNBUFFEREDbuffered_stdio
PYTHONVERBOSEverbose
PYTHONWARNINGSwarnoptions

PYTHONLEGACYWINDOWSFSENCODING andPYTHONLEGACYWINDOWSSTDIO arespecific to Windows.

Default Python Configuration

PyPreConfig_InitPythonConfig():

  • allocator =PYMEM_ALLOCATOR_NOT_SET
  • coerce_c_locale_warn = -1
  • coerce_c_locale = -1
  • configure_locale = 1
  • dev_mode = -1
  • isolated = 0
  • legacy_windows_fs_encoding = -1
  • use_environment = 1
  • utf8_mode = -1

PyConfig_InitPythonConfig():

  • argv = []
  • base_exec_prefix =NULL
  • base_prefix =NULL
  • buffered_stdio = 1
  • bytes_warning = 0
  • check_hash_pycs_mode =NULL
  • configure_c_stdio = 1
  • dev_mode = 0
  • dump_refs = 0
  • exec_prefix =NULL
  • executable =NULL
  • faulthandler = 0
  • filesystem_encoding =NULL
  • filesystem_errors =NULL
  • hash_seed = 0
  • home =NULL
  • import_time = 0
  • inspect = 0
  • install_signal_handlers = 1
  • interactive = 0
  • isolated = 0
  • malloc_stats = 0
  • module_search_path_env =NULL
  • module_search_paths = []
  • optimization_level = 0
  • parse_argv = 1
  • parser_debug = 0
  • pathconfig_warnings = 1
  • prefix =NULL
  • program_name =NULL
  • pycache_prefix =NULL
  • quiet = 0
  • run_command =NULL
  • run_filename =NULL
  • run_module =NULL
  • show_alloc_count = 0
  • show_ref_count = 0
  • site_import = 1
  • skip_source_first_line = 0
  • stdio_encoding =NULL
  • stdio_errors =NULL
  • tracemalloc = 0
  • use_environment = 1
  • use_hash_seed = 0
  • user_site_directory = 1
  • verbose = 0
  • warnoptions = []
  • write_bytecode = 1
  • xoptions = []
  • _init_main = 1
  • _install_importlib = 1

Default Isolated Configuration

PyPreConfig_InitIsolatedConfig():

  • allocator =PYMEM_ALLOCATOR_NOT_SET
  • coerce_c_locale_warn = 0
  • coerce_c_locale = 0
  • configure_locale = 0
  • dev_mode = 0
  • isolated = 1
  • legacy_windows_fs_encoding = 0
  • use_environment = 0
  • utf8_mode = 0

PyConfig_InitIsolatedConfig():

  • argv = []
  • base_exec_prefix =NULL
  • base_prefix =NULL
  • buffered_stdio = 1
  • bytes_warning = 0
  • check_hash_pycs_mode =NULL
  • configure_c_stdio = 0
  • dev_mode = 0
  • dump_refs = 0
  • exec_prefix =NULL
  • executable =NULL
  • faulthandler = 0
  • filesystem_encoding =NULL
  • filesystem_errors =NULL
  • hash_seed = 0
  • home =NULL
  • import_time = 0
  • inspect = 0
  • install_signal_handlers = 0
  • interactive = 0
  • isolated = 1
  • malloc_stats = 0
  • module_search_path_env =NULL
  • module_search_paths = []
  • optimization_level = 0
  • parse_argv = 0
  • parser_debug = 0
  • pathconfig_warnings = 0
  • prefix =NULL
  • program_name =NULL
  • pycache_prefix =NULL
  • quiet = 0
  • run_command =NULL
  • run_filename =NULL
  • run_module =NULL
  • show_alloc_count = 0
  • show_ref_count = 0
  • site_import = 1
  • skip_source_first_line = 0
  • stdio_encoding =NULL
  • stdio_errors =NULL
  • tracemalloc = 0
  • use_environment = 0
  • use_hash_seed = 0
  • user_site_directory = 0
  • verbose = 0
  • warnoptions = []
  • write_bytecode = 1
  • xoptions = []
  • _init_main = 1
  • _install_importlib = 1

Python 3.7 API

Python 3.7 has 4 functions in its C API to initialize and finalizePython:

  • Py_Initialize(),Py_InitializeEx(): initialize Python
  • Py_Finalize(),Py_FinalizeEx(): finalize Python

Python 3.7 can be configured usingGlobal Configuration Variables,Environment Variables, and the following functions:

  • PyImport_AppendInittab()
  • PyImport_ExtendInittab()
  • PyMem_SetAllocator()
  • PyMem_SetupDebugHooks()
  • PyObject_SetArenaAllocator()
  • Py_SetPath()
  • Py_SetProgramName()
  • Py_SetPythonHome()
  • Py_SetStandardStreamEncoding()
  • PySys_AddWarnOption()
  • PySys_AddXOption()
  • PySys_ResetWarnOptions()

There is also a high-levelPy_Main() function andPyImport_FrozenModules variable which can be overridden.

SeeInitialization, Finalization, and Threads documentation.

Python Issues

Issues that will be fixed by this PEP, directly or indirectly:

  • bpo-1195571: “simplecallback system for Py_FatalError”
  • bpo-11320:“Usage of API method Py_SetPath causes errors in Py_Initialize()(Posix only)”
  • bpo-13533: “Would likePy_Initialize to play friendly with host app”
  • bpo-14956: “custom PYTHONPATHmay break apps embedding Python”
  • bpo-19983: “When interruptedduring startup, Python should not call abort() but exit()”
  • bpo-22213: “Make pyvenv stylevirtual environments easier to configure when embedding Python”.
  • bpo-29778: “_Py_CheckPython3uses uninitialized dllpath when embedder sets module path withPy_SetPath”
  • bpo-30560: “AddPy_SetFatalErrorAbortFunc: Allow embedding program to handle fatalerrors”.
  • bpo-31745: “Overloading“Py_GetPath” does not work”
  • bpo-32573: “All sys attributes(.argv, …) should exist in embedded environments”.
  • bpo-33135: “Define fieldprefixes for the various config structs”. The PEP now defines wellhow warnings options are handled.
  • bpo-34725:“Py_GetProgramFullPath() odd behaviour in Windows”
  • bpo-36204: “Deprecate callingPy_Main() after Py_Initialize()? Add Py_InitializeFromArgv()?”

Issues of the PEP implementation:

  • bpo-16961: “No regressiontests for -E and individual environment vars”
  • bpo-20361: “-W command lineoptions and PYTHONWARNINGS environmental variable should not override-b / -bb command line options”
  • bpo-26122: “Isolated modedoesn’t ignore PYTHONHASHSEED”
  • bpo-29818:“Py_SetStandardStreamEncoding leads to a memory error in debug mode”
  • bpo-31845:“PYTHONDONTWRITEBYTECODE and PYTHONOPTIMIZE have no effect”
  • bpo-32030: “PEP 432: RewritePy_Main()”
  • bpo-32124: “Document functionssafe to be called before Py_Initialize()”
  • bpo-33042: “New 3.7 startupsequence crashes PyInstaller”
  • bpo-33932: “CallingPy_Initialize() twice now triggers a fatal error (Python 3.7)”
  • bpo-34008: “Do we supportcalling Py_Main() after Py_Initialize()?”
  • bpo-34170: “Py_Initialize():computing path configuration must not have side effect (PEP 432)”
  • bpo-34589: “Py_Initialize()and Py_Main() should not enable C locale coercion”
  • bpo-34639:“PYTHONCOERCECLOCALE is ignored when using -E or -I option”
  • bpo-36142: “Add a new_PyPreConfig step to Python initialization to setup memory allocatorand encodings”
  • bpo-36202: “CallingPy_DecodeLocale() before _PyPreConfig_Write() can produce mojibake”
  • bpo-36301: “Add_Py_PreInitialize() function”
  • bpo-36443: “Disablecoerce_c_locale and utf8_mode by default in _PyPreConfig?”
  • bpo-36444: “Pythoninitialization: remove _PyMainInterpreterConfig”
  • bpo-36471: “PEP 432, PEP 587:Add _Py_RunMain()”
  • bpo-36763: “PEP 587: Reworkinitialization API to prepare second version of the PEP”
  • bpo-36775: “Rework filesystemcodec implementation”
  • bpo-36900: “Use _PyCoreConfigrather than global configuration variables”

Issues related to this PEP:

  • bpo-12598: “Move sys variableinitialization from import.c to sysmodule.c”
  • bpo-15577: “Real argc and argvin embedded interpreter”
  • bpo-16202: “sys.path[0]security issues”
  • bpo-18309: “Make pythonslightly more relocatable”
  • bpo-22257: “PEP 432: Redesignthe interpreter startup sequence”
  • bpo-25631: “Segmentation faultwith invalid Unicode command-line arguments in embedded Python”
  • bpo-26007: “Support embeddingthe standard library in an executable”
  • bpo-31210: “Can not importmodules if sys.prefix contains DELIM”.
  • bpo-31349: “Embeddedinitialization ignores Py_SetProgramName()”
  • bpo-33919: “Expose_PyCoreConfig structure to Python”
  • bpo-35173: “Re-use alreadyexisting functionality to allow Python 2.7.x (both embedded andstandalone) to locate the module path according to the shared library”

Discussions

Version History

  • Version 5:
    • RenamePyInitError toPyStatus
    • RenamePyInitError_Failed() toPyStatus_Exception()
    • RenamePy_ExitInitError() toPy_ExitStatusException()
    • AddPyPreConfig._config_init private field.
    • Fix Python Configuration default values: isolated=0and use_environment=1, instead of -1.
    • Add “Multi-Phase Initialization Private Provisional API”and “Discussions” sections
  • Version 4:
    • Introduce “Python Configuration” and “Isolated Configuration” whichare well better defined. Replace all macros with functions.
    • ReplacePyPreConfig_INIT andPyConfig_INIT macros withfunctions:
      • PyPreConfig_InitIsolatedConfig(),PyConfig_InitIsolatedConfig()
      • PyPreConfig_InitPythonConfig(),PyConfig_InitPythonConfig()
    • PyPreConfig no longer uses dynamic memory, theallocatorfield type becomes an int, addconfigure_locale andparse_argv field.
    • PyConfig: renamemodule_search_path_env topythonpath_env, renameuse_module_search_paths tomodule_search_paths_set, removeprogram anddll_path.
    • ReplacePy_INIT_xxx() macros withPyInitError_xxx()functions.
    • Remove the “Constant PyConfig” section. RemovePy_InitializeFromArgs() andPy_InitializeFromBytesArgs()functions.
  • Version 3:
    • PyConfig: Addconfigure_c_stdio andparse_argv;rename_frozen topathconfig_warnings.
    • Rename functions using bytes strings and wide character strings. Forexample,Py_PreInitializeFromWideArgs() becomesPy_PreInitializeFromArgs(), andPyConfig_SetArgv() becomesPyConfig_SetBytesArgv().
    • AddPyWideStringList_Insert() function.
    • New “Path configuration”, “Isolate Python”, “Python Issues”and “Version History” sections.
    • PyConfig_SetString() andPyConfig_SetBytesString() nowrequires the configuration as the first argument.
    • RenamePy_UnixMain() toPy_BytesMain()
  • Version 2: AddPyConfig methods (ex:PyConfig_Read()), addPyWideStringList_Append(), renamePyWideCharList toPyWideStringList.
  • Version 1: Initial version.

Acceptance

PEP 587 wasaccepted by Thomas Wouters on May 26, 2019.

Copyright

This document has been placed in the public domain.


Source:https://github.com/python/peps/blob/main/peps/pep-0587.rst

Last modified:2025-02-01 08:55:40 GMT


[8]ページ先頭

©2009-2025 Movatter.jp