Add a new C API to configure the Python Initialization providing finercontrol on the whole configuration and better error reporting.
It becomes possible to read the configuration and then override somecomputed parameters before it is applied. It also becomes possible tocompletely override how Python computes the module search paths(sys.path).
The newIsolated Configuration provides sane default values toisolate Python from the system. For example, to embed Python into anapplication. Using the environment are now opt-in options, rather thanan opt-out options. For example, environment variables, command linearguments and global configuration variables are ignored by default.
Building a customized Python which behaves as the regular Python becomeseasier using the newPy_RunMain() function. Moreover, using thePython Configuration,PyConfig.argv arguments are now parsed thesame way the regular Python parses command line arguments, andPyConfig.xoptions are handled as-Xopt command line options.
This extracts a subset of the API design from thePEP 432 developmentand refactoring work that is now considered sufficiently stable to makepublic (allowing 3rd party embedding applications access to the sameconfiguration APIs that the native CPython CLI is now using).
Python is highly configurable but its configuration evolved organically.The initialization configuration is scattered all around the code usingdifferent ways to set them: global configuration variables (ex:Py_IsolatedFlag), environment variables (ex:PYTHONPATH),command line arguments (ex:-b), configuration files (ex:pyvenv.cfg), function calls (ex:Py_SetProgramName()). Astraightforward and reliable way to configure Python is needed.
Some configuration parameters are not accessible from the C API, or noteasily. For example, there is no API to override the default values ofsys.executable.
Some options likePYTHONPATH can only be set using an environmentvariable which has a side effect on Python child processes if not unsetproperly.
Some options also depends on other options: seePriority and Rules.Python 3.7 API does not provide a consistent view of the overallconfiguration.
The C API of Python 3.7 Initialization takeswchar_t* strings asinput whereas the Python filesystem encoding is set during theinitialization which can lead to mojibake.
Python 3.7 APIs likePy_Initialize() aborts the process on memoryallocation failure which is not convenient when Python is embedded.Moreover,Py_Main() could exit directly the process rather thanreturning an exit code. Proposed new API reports the error or exit codeto the caller which can decide how to handle it.
Implementing thePEP 540 (UTF-8 Mode) and the new-Xdev correctlywas almost impossible in Python 3.6. The code base has been deeplyreworked in Python 3.7 and then in Python 3.8 to read the configurationinto a structure with no side effect. It becomes possible to clear theconfiguration (release memory) and read again the configuration if theencoding changed . It is required to implement properly the UTF-8 whichchanges the encoding using-Xutf8 command line option. Internally,bytesargv strings are decoded from the filesystem encoding. The-Xdev changes the memory allocator (behaves asPYTHONMALLOC=debug), whereas it was not possible to change thememory allocationwhile parsing the command line arguments. The newdesign of the internal implementation not only allowed to implementproperly-Xutf8 and-Xdev, it also allows to change the Pythonbehavior way more easily, especially for corner cases like that, andensure that the configuration remains consistent: seePriority andRules.
This PEP is a partial implementation ofPEP 432 which is the overalldesign. New fields can be added later toPyConfig structure tofinish the implementation of thePEP 432 (e.g. by adding a new partialinitialization API which allows to configure Python using Python objects tofinish the full initialization). However, those features are omitted from thisPEP as even the native CPython CLI doesn’t work that way - the public APIproposal in this PEP is limited to features which have already been implementedand adopted as private APIs for us in the native CPython CLI.
This PEP proposes to add the following new structures and functions.
New structures:
PyConfigPyPreConfigPyStatusPyWideStringListNew functions:
PyConfig_Clear(config)PyConfig_InitIsolatedConfig()PyConfig_InitPythonConfig()PyConfig_Read(config)PyConfig_SetArgv(config,argc,argv)PyConfig_SetBytesArgv(config,argc,argv)PyConfig_SetBytesString(config,config_str,str)PyConfig_SetString(config,config_str,str)PyPreConfig_InitIsolatedConfig(preconfig)PyPreConfig_InitPythonConfig(preconfig)PyStatus_Error(err_msg)PyStatus_Exception(status)PyStatus_Exit(exitcode)PyStatus_IsError(status)PyStatus_IsExit(status)PyStatus_NoMemory()PyStatus_Ok()PyWideStringList_Append(list,item)PyWideStringList_Insert(list,index,item)Py_BytesMain(argc,argv)Py_ExitStatusException(status)Py_InitializeFromConfig(config)Py_PreInitialize(preconfig)Py_PreInitializeFromArgs(preconfig,argc,argv)Py_PreInitializeFromBytesArgs(preconfig,argc,argv)Py_RunMain()This PEP also adds_PyRuntimeState.preconfig (PyPreConfig type)andPyInterpreterState.config (PyConfig type) fields to theseinternal structures.PyInterpreterState.config becomes the newreference configuration, replacing global configuration variables andother private variables.
PyWideStringList is a list ofwchar_t* strings.
PyWideStringList structure fields:
length (Py_ssize_t)items (wchar_t**)Methods:
PyStatusPyWideStringList_Append(PyWideStringList*list,constwchar_t*item):Appenditem tolist.PyStatusPyWideStringList_Insert(PyWideStringList*list,Py_ssize_tindex,constwchar_t*item):Insertitem intolist atindex. Ifindex is greater thanlist length, just appenditem tolist.Iflength is non-zero,items must be non-NULL and all strings mustbe non-NULL.
PyStatus is a structure to store the status of an initializationfunction: success, error or exit. For an error, it can store the Cfunction name which created the error.
Example:
PyStatusalloc(void**ptr,size_tsize){*ptr=PyMem_RawMalloc(size);if(*ptr==NULL){returnPyStatus_NoMemory();}returnPyStatus_Ok();}intmain(intargc,char**argv){void*ptr;PyStatusstatus=alloc(&ptr,16);if(PyStatus_Exception(status)){Py_ExitStatusException(status);}PyMem_Free(ptr);return0;}
PyStatus fields:
exitcode (int):Argument passed toexit().err_msg (constchar*):Error message.func (constchar*):Name of the function which created an error, can beNULL._type field: for internal usage only.Functions to create a status:
PyStatus_Ok(): Success.PyStatus_Error(err_msg): Initialization error with a message.PyStatus_NoMemory(): Memory allocation failure (out of memory).PyStatus_Exit(exitcode): Exit Python with the specified exit code.Functions to handle a status:
PyStatus_Exception(status): Is the result an error or an exit?If true, the exception must be handled; by callingPy_ExitStatusException(status) for example.PyStatus_IsError(status): Is the result an error?PyStatus_IsExit(status): Is the result an exit?Py_ExitStatusException(status): Callexit(exitcode) ifstatusis an exit. Print the error messageand exit with a non-zero exit codeifstatus is an error. Must only be called ifPyStatus_Exception(status) is true.Note
Internally, Python uses macros which setPyStatus.func,whereas functions to create a status setfunc toNULL.
ThePyPreConfig structure is used to preinitialize Python:
Example using the preinitialization to enable the UTF-8 Mode:
PyStatusstatus;PyPreConfigpreconfig;PyPreConfig_InitPythonConfig(&preconfig);preconfig.utf8_mode=1;status=Py_PreInitialize(&preconfig);if(PyStatus_Exception(status)){Py_ExitStatusException(status);}/*atthispoint,PythonwillspeakUTF-8*/Py_Initialize();/*...usePythonAPIhere...*/Py_Finalize();
Function to initialize a preconfiguration:
PyStatusPyPreConfig_InitIsolatedConfig(PyPreConfig*preconfig)PyStatusPyPreConfig_InitPythonConfig(PyPreConfig*preconfig)Functions to preinitialize Python:
PyStatusPy_PreInitialize(constPyPreConfig*preconfig)PyStatusPy_PreInitializeFromBytesArgs(constPyPreConfig*preconfig,intargc,char*const*argv)PyStatusPy_PreInitializeFromArgs(constPyPreConfig*preconfig,intargc,wchar_t*const*argv)The caller is responsible to handle exceptions (error or exit) usingPyStatus_Exception() andPy_ExitStatusException().
ForPython Configuration (PyPreConfig_InitPythonConfig()),if Python is initialized with command line arguments, the command linearguments must also be passed to preinitialize Python, since they havean effect on the pre-configuration like encodings. For example, the-Xutf8 command line option enables the UTF-8 Mode.
PyPreConfig fields:
allocator (int):Name of the memory allocator (ex:PYMEM_ALLOCATOR_MALLOC). Validvalues:PYMEM_ALLOCATOR_NOT_SET (0): don’t change memory allocators(use defaults)PYMEM_ALLOCATOR_DEFAULT (1): default memory allocatorsPYMEM_ALLOCATOR_DEBUG (2): default memory allocators withdebug hooksPYMEM_ALLOCATOR_MALLOC (3): force usage ofmalloc()PYMEM_ALLOCATOR_MALLOC_DEBUG (4): force usage ofmalloc() with debug hooksPYMEM_ALLOCATOR_PYMALLOC (5): Python “pymalloc” allocatorPYMEM_ALLOCATOR_PYMALLOC_DEBUG (6): Python “pymalloc”allocator with debug hooksPYMEM_ALLOCATOR_PYMALLOC andPYMEM_ALLOCATOR_PYMALLOC_DEBUG are not supported if Python isconfigured using--without-pymallocconfigure_locale (int):Set the LC_CTYPE locale to the user preferred locale? If equals to 0,setcoerce_c_locale andcoerce_c_locale_warn to 0.coerce_c_locale (int):If equals to 2, coerce the C locale; if equals to 1, read the LC_CTYPElocale to decide if it should be coerced.coerce_c_locale_warn (int):If non-zero, emit a warning if the C locale is coerced.dev_mode (int):SeePyConfig.dev_mode.isolated (int):SeePyConfig.isolated.legacy_windows_fs_encoding (int, Windows only):If non-zero, disable UTF-8 Mode, set the Python filesystem encoding tombcs, set the filesystem error handler toreplace.parse_argv (int):If non-zero,Py_PreInitializeFromArgs() andPy_PreInitializeFromBytesArgs() parse theirargv argument thesame way the regular Python parses command line arguments: seeCommand Line Arguments.use_environment (int):SeePyConfig.use_environment.utf8_mode (int):If non-zero, enable the UTF-8 mode.Thelegacy_windows_fs_encoding field is only available on Windows.#ifdefMS_WINDOWS macro can be used for Windows specific code.
PyPreConfig private fields, for internal use only:
_config_init (int):Function used to initializePyConfig, used for preinitialization.PyMem_SetAllocator() can be called afterPy_PreInitialize() andbeforePy_InitializeFromConfig() to install a custom memoryallocator. It can be called beforePy_PreInitialize() ifallocator is set toPYMEM_ALLOCATOR_NOT_SET (default value).
Python memory allocation functions likePyMem_RawMalloc() must notbe used before Python preinitialization, whereas calling directlymalloc() andfree() is always safe.Py_DecodeLocale() mustnot be called before the preinitialization.
ThePyConfig structure contains most parameters to configure Python.
Example setting the program name:
voidinit_python(void){PyStatusstatus;PyConfigconfig;PyConfig_InitPythonConfig(&config);/*Settheprogramname.ImplicitlypreinitializePython.*/status=PyConfig_SetString(&config,&config.program_name,L"/path/to/my_program");if(PyStatus_Exception(status)){gotofail;}status=Py_InitializeFromConfig(&config);if(PyStatus_Exception(status)){gotofail;}PyConfig_Clear(&config);return;fail:PyConfig_Clear(&config);Py_ExitStatusException(status);}
PyConfig methods:
voidPyConfig_InitPythonConfig(PyConfig*config)Initialize configuration withPython Configuration.voidPyConfig_InitIsolatedConfig(PyConfig*config):Initialize configuration withIsolated Configuration.PyStatusPyConfig_SetString(PyConfig*config,wchar_t*const*config_str,constwchar_t*str):Copy the wide character stringstr into*config_str.Preinitialize Python if needed.PyStatusPyConfig_SetBytesString(PyConfig*config,wchar_t*const*config_str,constchar*str):Decodestr usingPy_DecodeLocale() and set the result into*config_str.Preinitialize Python if needed.PyStatusPyConfig_SetArgv(PyConfig*config,intargc,wchar_t*const*argv):Set command line arguments from wide character strings.Preinitialize Python if needed.PyStatusPyConfig_SetBytesArgv(PyConfig*config,intargc,char*const*argv):Set command line arguments: decode bytes usingPy_DecodeLocale().Preinitialize Python if needed.PyStatusPyConfig_Read(PyConfig*config):Read all Python configuration. Fields which are already initializedare left unchanged.Preinitialize Python if needed.voidPyConfig_Clear(PyConfig*config):Release configuration memory.MostPyConfig methods preinitialize Python if needed. In that case,the Python preinitialization configuration in based on thePyConfig.If configuration fields which are in common withPyPreConfig aretuned, they must be set before calling aPyConfig method:
dev_modeisolatedparse_argvuse_environmentMoreover, ifPyConfig_SetArgv() orPyConfig_SetBytesArgv() isused, this method must be called first, before other methods, since thepreinitialization configuration depends on command line arguments (ifparse_argv is non-zero).
Functions to initialize Python:
PyStatusPy_InitializeFromConfig(constPyConfig*config):Initialize Python fromconfig configuration.The caller of these methods and functions is responsible to handleexceptions (error or exit) usingPyStatus_Exception() andPy_ExitStatusException().
PyConfig fields:
argv (PyWideStringList):Command line arguments,sys.argv. Seeparse_argv to parseargv the same way the regular Python parses Python command linearguments. Ifargv is empty, an empty string is added to ensurethatsys.argv always exists and is never empty.base_exec_prefix (wchar_t*):sys.base_exec_prefix.base_prefix (wchar_t*):sys.base_prefix.buffered_stdio (int):If equals to 0, enable unbuffered mode, making the stdout and stderr streamsunbuffered.bytes_warning (int):If equals to 1, issue a warning when comparingbytes orbytearray withstr, or comparingbytes withint. Ifequal or greater to 2, raise aBytesWarning exception.check_hash_pycs_mode (wchar_t*):--check-hash-based-pycs command line option value (seePEP 552).Valid values:always,never anddefault. The default valueisdefault.configure_c_stdio (int):If non-zero, configure C standard streams (stdio,stdout,stdout). For example, set their mode toO_BINARY on Windows.dev_mode (int):Development modedump_refs (int):If non-zero, dump all objects which are still alive at exit.Require a special Python build withPy_REF_DEBUG macro defined.exec_prefix (wchar_t*):sys.exec_prefix.executable (wchar_t*):sys.executable.faulthandler (int):If non-zero, callfaulthandler.enable().filesystem_encoding (wchar_t*):Filesystem encoding,sys.getfilesystemencoding().filesystem_errors (wchar_t*):Filesystem encoding errors,sys.getfilesystemencodeerrors().use_hash_seed (int),hash_seed (unsignedlong):Randomized hash function seed.home (wchar_t*):Python home directory.import_time (int):If non-zero, profile import time.inspect (int):Enter interactive mode after executing a script or a command.install_signal_handlers (int):Install signal handlers?interactive (int):Interactive mode.isolated (int):If greater than 0, enable isolated mode:sys.path contains neither the script’s directory (computed fromargv[0] or the current directory) nor the user’s site-packagesdirectory.readline nor enable default readlineconfiguration on interactive prompts.use_environment anduser_site_directory to 0.legacy_windows_stdio (int, Windows only):If non-zero, useio.FileIO instead ofWindowsConsoleIO forsys.stdin,sys.stdout andsys.stderr.malloc_stats (int):If non-zero, dump statistics onpymalloc memory allocator at exit.The option is ignored if Python is built using--without-pymalloc.pythonpath_env (wchar_t*):Module search paths as a string separated by DELIM (usually:character).Initialized fromPYTHONPATH environment variable value by default.module_search_paths_set (int),module_search_paths (PyWideStringList):sys.path. Ifmodule_search_paths_set is equal to 0, themodule_search_paths is overridden by the function computing thePath Configuration.optimization_level (int):Compilation optimization level:__debug__ is set toTrue)__debug__ toFalseparse_argv (int):If non-zero, parseargv the same way the regular Python commandline arguments, and strip Python arguments fromargv: seeCommandLine Arguments.parser_debug (int):If non-zero, turn on parser debugging output (for expert only,depending on compilation options).pathconfig_warnings (int):If equal to 0, suppress warnings when computing the pathconfiguration (Unix only, Windows does not log any warning). Otherwise,warnings are written into stderr.prefix (wchar_t*):sys.prefix.program_name (wchar_t*):Program name.pycache_prefix (wchar_t*):.pyc cache prefix.quiet (int):Quiet mode. For example, don’t display the copyright and versionmessages even in interactive mode.run_command (wchar_t*):python3-cCOMMAND argument.run_filename (wchar_t*):python3FILENAME argument.run_module (wchar_t*):python3-mMODULE argument.show_alloc_count (int):Show allocation counts at exit?Need a special Python build withCOUNT_ALLOCS macro defined.show_ref_count (int):Show total reference count at exit?Need a debug build of Python (Py_REF_DEBUG macro should bedefined).site_import (int):Import thesite module at startup?skip_source_first_line (int):Skip the first line of the source?stdio_encoding (wchar_t*),stdio_errors (wchar_t*):Encoding and encoding errors ofsys.stdin,sys.stdoutandsys.stderr.tracemalloc (int):If non-zero, calltracemalloc.start(value).user_site_directory (int):If non-zero, add user site directory tosys.path.verbose (int):If non-zero, enable verbose mode.warnoptions (PyWideStringList):Options of thewarnings module to build warnings filters.write_bytecode (int):If non-zero, write.pyc files.xoptions (PyWideStringList):sys._xoptions.Thelegacy_windows_stdio field is only available on Windows.#ifdefMS_WINDOWS macro can be used for Windows specific code.
Ifparse_argv is non-zero,argv arguments are parsed the sameway the regular Python parses command line arguments, and Pythonarguments are stripped fromargv: seeCommand Line Arguments.
Thexoptions options are parsed to set other options: see-XOptions.
PyConfig private fields, for internal use only:
_config_init (int):Function used to initializePyConfig, used for preinitialization._install_importlib (int):Install importlib?_init_main (int):If equal to 0, stop Python initialization before the “main” phase(seePEP 432).More complete example modifying the default configuration, read theconfiguration, and then override some parameters:
PyStatusinit_python(constchar*program_name){PyStatusstatus;PyConfigconfig;PyConfig_InitPythonConfig(&config);/*Settheprogramnamebeforereadingtheconfiguration(decodebytestringfromthelocaleencoding).ImplicitlypreinitializePython.*/status=PyConfig_SetBytesString(&config,&config.program_name,program_name);if(PyStatus_Exception(status)){gotodone;}/*Readallconfigurationatonce*/status=PyConfig_Read(&config);if(PyStatus_Exception(status)){gotodone;}/*Appendourcustomsearchpathtosys.path*/status=PyWideStringList_Append(&config.module_search_paths,L"/path/to/more/modules");if(PyStatus_Exception(status)){gotodone;}/*OverrideexecutablecomputedbyPyConfig_Read()*/status=PyConfig_SetString(&config,&config.executable,L"/path/to/my_executable");if(PyStatus_Exception(status)){gotodone;}status=Py_InitializeFromConfig(&config);done:PyConfig_Clear(&config);returnstatus;}
Note
PyImport_FrozenModules,PyImport_AppendInittab() andPyImport_ExtendInittab() functions are still relevant andcontinue to work as previously. They should be set or called afterPython preinitialization and before the Python initialization.
PyPreConfig_InitIsolatedConfig() andPyConfig_InitIsolatedConfig() functions create a configuration toisolate Python from the system. For example, to embed Python into anapplication.
This configuration ignores global configuration variables, environmentsvariables and command line arguments (argv is not parsed). The Cstandard streams (ex:stdout) and the LC_CTYPE locale are leftunchanged by default.
Configuration files are still used with this configuration. Set thePath Configuration (“output fields”) to ignore these configurationfiles and avoid the function computing the default path configuration.
PyPreConfig_InitPythonConfig() andPyConfig_InitPythonConfig()functions create a configuration to build a customized Python whichbehaves as the regular Python.
Environments variables and command line arguments are used to configurePython, whereas global configuration variables are ignored.
This function enables C locale coercion (PEP 538) and UTF-8 Mode (PEP540) depending on the LC_CTYPE locale,PYTHONUTF8 andPYTHONCOERCECLOCALE environment variables.
Example of customized Python always running in isolated mode:
intmain(intargc,char**argv){PyStatusstatus;PyConfigconfig;PyConfig_InitPythonConfig(&config);config.isolated=1;/*Decodecommandlinearguments.ImplicitlypreinitializePython(inisolatedmode).*/status=PyConfig_SetBytesArgv(&config,argc,argv);if(PyStatus_Exception(status)){gotofail;}status=Py_InitializeFromConfig(&config);if(PyStatus_Exception(status)){gotofail;}PyConfig_Clear(&config);returnPy_RunMain();fail:PyConfig_Clear(&config);if(PyStatus_IsExit(status)){returnstatus.exitcode;}/*Displaytheerrormessageandexittheprocesswithnon-zeroexitcode*/Py_ExitStatusException(status);}
This example is a basic implementation of the “System Python Executable”discussed inPEP 432.
PyConfig contains multiple fields for the path configuration:
homepythonpath_envpathconfig_warningsexec_prefixexecutableprefixmodule_search_paths_set,module_search_pathsIf at least one “output field” is not set, Python computes the pathconfiguration to fill unset fields. Ifmodule_search_paths_set isequal to 0,module_search_paths is overridden andmodule_search_paths_set is set to 1.
It is possible to completely ignore the function computing the defaultpath configuration by setting explicitly all path configuration outputfields listed above. A string is considered as set even if it is non-empty.module_search_paths is considered as set ifmodule_search_paths_set is set to 1. In this case, pathconfiguration input fields are ignored as well.
Setpathconfig_warnings to 0 to suppress warnings when computing thepath configuration (Unix only, Windows does not log any warning).
Ifbase_prefix orbase_exec_prefix fields are not set, theyinherit their value fromprefix andexec_prefix respectively.
Py_RunMain() andPy_Main() modifysys.path:
run_filename is set and is a directory which contains a__main__.py script, prependrun_filename tosys.path.isolated is zero:run_module is set, prepend the current directory tosys.path. Do nothing if the current directory cannot be read.run_filename is set, prepends the directory of the filenametosys.path.sys.path.Ifsite_import is non-zero,sys.path can be modified by thesite module. Ifuser_site_directory is non-zero and the user’ssite-package directory exists, thesite module appends the user’ssite-package directory tosys.path.
See alsoConfiguration Files used by the path configuration.
Python 3.7 provides a high-levelPy_Main() function which requiresto pass command line arguments aswchar_t* strings. It isnon-trivial to use the correct encoding to decode bytes. Python has itsown set of issues with C locale coercion and UTF-8 Mode.
This PEP adds a newPy_BytesMain() function which takes command linearguments as bytes:
intPy_BytesMain(intargc,char**argv)
The newPy_RunMain() function executes the command(PyConfig.run_command), the script (PyConfig.run_filename) orthe module (PyConfig.run_module) specified on the command line or inthe configuration, and then finalizes Python. It returns an exit statusthat can be passed to theexit() function.
intPy_RunMain(void);
SeePython Configuration for an example of customized Python alwaysrunning in isolated mode usingPy_RunMain().
This section is a private provisional API introducing multi-phaseinitialization, the core feature of thePEP 432:
sys module is only partially initialized(ex:sys.path doesn’t exist yet);importlib;sys module initialization (ex: createsys.stdout andsys.path);faulthandler andtracemalloc;site module;Private provisional API:
PyConfig._init_main: if set to 0,Py_InitializeFromConfig()stops at the “Core” initialization phase.PyStatus_Py_InitializeMain(void): move to the “Main”initialization phase, finish the Python initialization.No module is imported during the “Core” phase and theimportlibmodule is not configured: thePath Configuration is only appliedduring the “Main” phase. It may allow to customize Python in Python tooverride or tune thePath Configuration, maybe install a customsys.meta_path importer or an import hook, etc.
It may become possible to compute thePath Configuration in Python,after the Core phase and before the Main phase, which is one of the PEP432 motivation.
The “Core” phase is not properly defined: what should be and what shouldnot be available at this phase is not specified yet. The API is markedas private and provisional: the API can be modified or even be removedanytime until a proper public API is designed.
Example running Python code between “Core” and “Main” initializationphases:
voidinit_python(void){PyStatusstatus;PyConfigconfig;PyConfig_InitPythonConfig(&config);config._init_main=0;/*...customize'config'configuration...*/status=Py_InitializeFromConfig(&config);PyConfig_Clear(&config);if(PyStatus_Exception(status)){Py_ExitStatusException(status);}/*Usesys.stderrbecausesys.stdoutisonlycreatedby_Py_InitializeMain()*/intres=PyRun_SimpleString("import sys; ""print('Run Python code before _Py_InitializeMain', ""file=sys.stderr)");if(res<0){exit(1);}/*...putmoreconfigurationcodehere...*/status=_Py_InitializeMain();if(PyStatus_Exception(status)){Py_ExitStatusException(status);}}
This PEP only adds a new API: it leaves the existing API unchanged andhas no impact on the backwards compatibility.
The Python 3.7Py_Initialize() function now disable the C localecoercion (PEP 538) and the UTF-8 Mode (PEP 540) by default to preventmojibake. The new API using thePython Configuration is needed toenable them automatically.
Differences betweenPyPreConfig_InitPythonConfig()andPyPreConfig_InitIsolatedConfig():
| PyPreConfig | Python | Isolated |
|---|---|---|
coerce_c_locale_warn | -1 | 0 |
coerce_c_locale | -1 | 0 |
configure_locale | 1 | 0 |
dev_mode | -1 | 0 |
isolated | 0 | 1 |
legacy_windows_fs_encoding | -1 | 0 |
use_environment | 0 | 0 |
parse_argv | 1 | 0 |
utf8_mode | -1 | 0 |
Differences betweenPyConfig_InitPythonConfig()andPyConfig_InitIsolatedConfig():
| PyConfig | Python | Isolated |
|---|---|---|
configure_c_stdio | 1 | 0 |
install_signal_handlers | 1 | 0 |
isolated | 0 | 1 |
parse_argv | 1 | 0 |
pathconfig_warnings | 1 | 0 |
use_environment | 1 | 0 |
user_site_directory | 1 | 0 |
Priority of configuration parameters, highest to lowest:
PyConfigPyPreConfigPriority of warning options, highest to lowest:
PyConfig.warnoptionsPySys_AddWarnOption()PyConfig.bytes_warning (add"error::BytesWarning" filter ifgreater than 1, add"default::BytesWarning filter if equals to 1)-Wopt command line argumentPYTHONWARNINGS environment variablePyConfig.dev_mode (add"default" filter)Rules onPyConfig parameters:
isolated is non-zero,use_environment anduser_site_directory are set to 0.dev_mode is non-zero,allocator is set to"debug",faulthandler is set to 1, and"default" filter is added towarnoptions. But thePYTHONMALLOC environment variable has thepriority overdev_mode to set the memory allocator.base_prefix is not set, it inheritsprefix value.base_exec_prefix is not set, it inheritsexec_prefix value.python._pth configuration file is present,isolated isset to 1 andsite_import is set to 0; butsite_import is setto 1 ifpython._pth containsimportsite.Rules onPyConfig andPyPreConfig parameters:
PyPreConfig.legacy_windows_fs_encoding is non-zero,setPyPreConfig.utf8_mode to 0, setPyConfig.filesystem_encoding tombcs, and setPyConfig.filesystem_errors toreplace.Python configuration files used by thePath Configuration:
pyvenv.cfgpython._pth (Windows only)pybuilddir.txt (Unix only)Global configuration variables mapped toPyPreConfig fields:
| Variable | Field |
|---|---|
Py_IgnoreEnvironmentFlag | use_environment (NOT) |
Py_IsolatedFlag | isolated |
Py_LegacyWindowsFSEncodingFlag | legacy_windows_fs_encoding |
Py_UTF8Mode | utf8_mode |
(NOT) means that thePyPreConfig value is the opposite of the globalconfiguration variable value.Py_LegacyWindowsFSEncodingFlag is onlyavailable on Windows.
Global configuration variables mapped toPyConfig fields:
| Variable | Field |
|---|---|
Py_BytesWarningFlag | bytes_warning |
Py_DebugFlag | parser_debug |
Py_DontWriteBytecodeFlag | write_bytecode (NOT) |
Py_FileSystemDefaultEncodeErrors | filesystem_errors |
Py_FileSystemDefaultEncoding | filesystem_encoding |
Py_FrozenFlag | pathconfig_warnings (NOT) |
Py_HasFileSystemDefaultEncoding | filesystem_encoding |
Py_HashRandomizationFlag | use_hash_seed,hash_seed |
Py_IgnoreEnvironmentFlag | use_environment (NOT) |
Py_InspectFlag | inspect |
Py_InteractiveFlag | interactive |
Py_IsolatedFlag | isolated |
Py_LegacyWindowsStdioFlag | legacy_windows_stdio |
Py_NoSiteFlag | site_import (NOT) |
Py_NoUserSiteDirectory | user_site_directory (NOT) |
Py_OptimizeFlag | optimization_level |
Py_QuietFlag | quiet |
Py_UnbufferedStdioFlag | buffered_stdio (NOT) |
Py_VerboseFlag | verbose |
_Py_HasFileSystemDefaultEncodeErrors | filesystem_errors |
(NOT) means that thePyConfig value is the opposite of the globalconfiguration variable value.Py_LegacyWindowsStdioFlag is onlyavailable on Windows.
Usage:
python3[options]python3[options]-cCOMMANDpython3[options]-mMODULEpython3[options]SCRIPT
Command line options mapped to pseudo-action onPyPreConfig fields:
| Option | PyConfig field |
|---|---|
-E | use_environment=0 |
-I | isolated=1 |
-Xdev | dev_mode=1 |
-Xutf8 | utf8_mode=1 |
-Xutf8=VALUE | utf8_mode=VALUE |
Command line options mapped to pseudo-action onPyConfig fields:
| Option | PyConfig field |
|---|---|
-b | bytes_warning++ |
-B | write_bytecode=0 |
-cCOMMAND | run_command=COMMAND |
--check-hash-based-pycs=MODE | check_hash_pycs_mode=MODE |
-d | parser_debug++ |
-E | use_environment=0 |
-i | inspect++ andinteractive++ |
-I | isolated=1 |
-mMODULE | run_module=MODULE |
-O | optimization_level++ |
-q | quiet++ |
-R | use_hash_seed=0 |
-s | user_site_directory=0 |
-S | site_import |
-t | ignored (kept for backwards compatibility) |
-u | buffered_stdio=0 |
-v | verbose++ |
-WWARNING | addWARNING towarnoptions |
-x | skip_source_first_line=1 |
-XOPTION | addOPTION toxoptions |
-h,-? and-V options are handled withoutPyConfig.
-X options mapped to pseudo-action onPyConfig fields:
| Option | PyConfig field |
|---|---|
-Xdev | dev_mode=1 |
-Xfaulthandler | faulthandler=1 |
-Ximporttime | import_time=1 |
-Xpycache_prefix=PREFIX | pycache_prefix=PREFIX |
-Xshowalloccount | show_alloc_count=1 |
-Xshowrefcount | show_ref_count=1 |
-Xtracemalloc=N | tracemalloc=N |
Environment variables mapped toPyPreConfig fields:
| Variable | PyPreConfig field |
|---|---|
PYTHONCOERCECLOCALE | coerce_c_locale,coerce_c_locale_warn |
PYTHONDEVMODE | dev_mode |
PYTHONLEGACYWINDOWSFSENCODING | legacy_windows_fs_encoding |
PYTHONMALLOC | allocator |
PYTHONUTF8 | utf8_mode |
Environment variables mapped toPyConfig fields:
| Variable | PyConfig field |
|---|---|
PYTHONDEBUG | parser_debug |
PYTHONDEVMODE | dev_mode |
PYTHONDONTWRITEBYTECODE | write_bytecode |
PYTHONDUMPREFS | dump_refs |
PYTHONEXECUTABLE | program_name |
PYTHONFAULTHANDLER | faulthandler |
PYTHONHASHSEED | use_hash_seed,hash_seed |
PYTHONHOME | home |
PYTHONINSPECT | inspect |
PYTHONIOENCODING | stdio_encoding,stdio_errors |
PYTHONLEGACYWINDOWSSTDIO | legacy_windows_stdio |
PYTHONMALLOCSTATS | malloc_stats |
PYTHONNOUSERSITE | user_site_directory |
PYTHONOPTIMIZE | optimization_level |
PYTHONPATH | pythonpath_env |
PYTHONPROFILEIMPORTTIME | import_time |
PYTHONPYCACHEPREFIX, | pycache_prefix |
PYTHONTRACEMALLOC | tracemalloc |
PYTHONUNBUFFERED | buffered_stdio |
PYTHONVERBOSE | verbose |
PYTHONWARNINGS | warnoptions |
PYTHONLEGACYWINDOWSFSENCODING andPYTHONLEGACYWINDOWSSTDIO arespecific to Windows.
PyPreConfig_InitPythonConfig():
allocator =PYMEM_ALLOCATOR_NOT_SETcoerce_c_locale_warn = -1coerce_c_locale = -1configure_locale = 1dev_mode = -1isolated = 0legacy_windows_fs_encoding = -1use_environment = 1utf8_mode = -1PyConfig_InitPythonConfig():
argv = []base_exec_prefix =NULLbase_prefix =NULLbuffered_stdio = 1bytes_warning = 0check_hash_pycs_mode =NULLconfigure_c_stdio = 1dev_mode = 0dump_refs = 0exec_prefix =NULLexecutable =NULLfaulthandler = 0filesystem_encoding =NULLfilesystem_errors =NULLhash_seed = 0home =NULLimport_time = 0inspect = 0install_signal_handlers = 1interactive = 0isolated = 0malloc_stats = 0module_search_path_env =NULLmodule_search_paths = []optimization_level = 0parse_argv = 1parser_debug = 0pathconfig_warnings = 1prefix =NULLprogram_name =NULLpycache_prefix =NULLquiet = 0run_command =NULLrun_filename =NULLrun_module =NULLshow_alloc_count = 0show_ref_count = 0site_import = 1skip_source_first_line = 0stdio_encoding =NULLstdio_errors =NULLtracemalloc = 0use_environment = 1use_hash_seed = 0user_site_directory = 1verbose = 0warnoptions = []write_bytecode = 1xoptions = []_init_main = 1_install_importlib = 1PyPreConfig_InitIsolatedConfig():
allocator =PYMEM_ALLOCATOR_NOT_SETcoerce_c_locale_warn = 0coerce_c_locale = 0configure_locale = 0dev_mode = 0isolated = 1legacy_windows_fs_encoding = 0use_environment = 0utf8_mode = 0PyConfig_InitIsolatedConfig():
argv = []base_exec_prefix =NULLbase_prefix =NULLbuffered_stdio = 1bytes_warning = 0check_hash_pycs_mode =NULLconfigure_c_stdio = 0dev_mode = 0dump_refs = 0exec_prefix =NULLexecutable =NULLfaulthandler = 0filesystem_encoding =NULLfilesystem_errors =NULLhash_seed = 0home =NULLimport_time = 0inspect = 0install_signal_handlers = 0interactive = 0isolated = 1malloc_stats = 0module_search_path_env =NULLmodule_search_paths = []optimization_level = 0parse_argv = 0parser_debug = 0pathconfig_warnings = 0prefix =NULLprogram_name =NULLpycache_prefix =NULLquiet = 0run_command =NULLrun_filename =NULLrun_module =NULLshow_alloc_count = 0show_ref_count = 0site_import = 1skip_source_first_line = 0stdio_encoding =NULLstdio_errors =NULLtracemalloc = 0use_environment = 0use_hash_seed = 0user_site_directory = 0verbose = 0warnoptions = []write_bytecode = 1xoptions = []_init_main = 1_install_importlib = 1Python 3.7 has 4 functions in its C API to initialize and finalizePython:
Py_Initialize(),Py_InitializeEx(): initialize PythonPy_Finalize(),Py_FinalizeEx(): finalize PythonPython 3.7 can be configured usingGlobal Configuration Variables,Environment Variables, and the following functions:
PyImport_AppendInittab()PyImport_ExtendInittab()PyMem_SetAllocator()PyMem_SetupDebugHooks()PyObject_SetArenaAllocator()Py_SetPath()Py_SetProgramName()Py_SetPythonHome()Py_SetStandardStreamEncoding()PySys_AddWarnOption()PySys_AddXOption()PySys_ResetWarnOptions()There is also a high-levelPy_Main() function andPyImport_FrozenModules variable which can be overridden.
SeeInitialization, Finalization, and Threads documentation.
Issues that will be fixed by this PEP, directly or indirectly:
Issues of the PEP implementation:
Issues related to this PEP:
PyInitError toPyStatusPyInitError_Failed() toPyStatus_Exception()Py_ExitInitError() toPy_ExitStatusException()PyPreConfig._config_init private field.PyPreConfig_INIT andPyConfig_INIT macros withfunctions:PyPreConfig_InitIsolatedConfig(),PyConfig_InitIsolatedConfig()PyPreConfig_InitPythonConfig(),PyConfig_InitPythonConfig()PyPreConfig no longer uses dynamic memory, theallocatorfield type becomes an int, addconfigure_locale andparse_argv field.PyConfig: renamemodule_search_path_env topythonpath_env, renameuse_module_search_paths tomodule_search_paths_set, removeprogram anddll_path.Py_INIT_xxx() macros withPyInitError_xxx()functions.Py_InitializeFromArgs() andPy_InitializeFromBytesArgs()functions.PyConfig: Addconfigure_c_stdio andparse_argv;rename_frozen topathconfig_warnings.Py_PreInitializeFromWideArgs() becomesPy_PreInitializeFromArgs(), andPyConfig_SetArgv() becomesPyConfig_SetBytesArgv().PyWideStringList_Insert() function.PyConfig_SetString() andPyConfig_SetBytesString() nowrequires the configuration as the first argument.Py_UnixMain() toPy_BytesMain()PyConfig methods (ex:PyConfig_Read()), addPyWideStringList_Append(), renamePyWideCharList toPyWideStringList.This document has been placed in the public domain.
Source:https://github.com/python/peps/blob/main/peps/pep-0587.rst
Last modified:2025-02-01 08:55:40 GMT