- Notifications
You must be signed in to change notification settings - Fork1
Python library for loading settings and config data from files and environment variables
License
daviskirk/climatecontrol
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
CLIMATECONTROL controls your applications settings and configurationenvironment. It is a Python library for loading app configurations from filesand/or namespaced environment variables.
- Separation of settings and code
- Loading from files (.yaml, .json, .toml)
- Loading multiple files using glob syntax
- Loading from environment variables, including loading of nested values
- Freely reference nested configurations via files or environment variables
- CLI integration
- Validation using the Validation library of your choice
- Logging configuration integration
- Testing integration
pip install climatecontrol
Set some environment variables in your shell
export CLIMATECONTROL_VALUE1=test1export CLIMATECONTROL_VALUE2=test2
Then use them in your python modules:
fromclimatecontrolimportclimateprint(climate.settings){'value1':'test1','value2':'test2'}
In case you want to update your settings or your environment variables havechanged and you want to reload them, the update method will reload yoursettings:
importosos.environ['CLIMATECONTROL_VALUE3']='new_env_data'climate.reload()print(climate.settings){'value1':'test1','value2':'test2','value3':'new_env_data'}
Now you've noticed that you want more complex configurations and need nestedsettings. For this situation we can delimit sections using a double underscore:
export CLIMATECONTROL_SECTION1__VALUE1=test1export CLIMATECONTROL_SECTION2__VALUE2=test2export CLIMATECONTROL_SECTION2__VALUE3=test3export CLIMATECONTROL_SECTION2__SUB_SECTION__VALUE4=test4
fromclimatecontrolimportclimateprint(climate.settings){'section1': {'value1':'test1' },'section2': {'value2':'test2','value3':'test3','sub_section': {'value4':'test4' } }}
If you don't want to use an environment variable for every single setting andwant to put your settings in a single file instead you can to this as well.Settings files can be yaml files (.yml/ .yaml), json files (.json) ortoml files (.toml).
export CLIMATECONTROL_SETTINGS_FILE=./my_settings_file.ymlThe file could look like this:
# ./climatecontrol_settings.yamlsection1:subsection1 = test1section2:subsection2:test2subsection3:test3
or in toml form:
# ./climatecontrol_settings.toml[section1]subsection1 ="test1"[section2]subsection2 ="test2"subsection3 ="test3"
In the following documentation examples, yaml files will be used, but anyexamples will work using the other file syntaxes as well.
See the climatecontrol.core.Climate.inferred_settings_files docstringfor further examples of how settings files are loaded and how they can be named.Also note that you can set your own settings files explicitely either bysettings an environment variable:
export CLIMATECONTROL_SETTINGS_FILE="mysettings.yaml, mysettings.toml, override.yml"
or by adding them in code:
climate.settings_files.extend(["mysettings.yaml","mysettings.toml","override.yml"]
Sometimes we don't want to save values in plain text in environment files or inthe settings file itself. Instead we have a file that contains the value of thesetting we want. A good example for this behaviour are dockersecrets thatstore secrets in temporary files.
To read a variable from a file, simply add a "_from_file" to the variablename and give it the path to the file that contains the variable as a value.
Using a settings file with the contents (in this case yaml):
section1:subsection1_from_file:/home/myuser/supersecret.txt
or using an environment variable:
export CLIMATECONTROL_SECTION1_SUBSECTION1_FROM_FILE="/home/myuser/supersecret.txt"
will both write the content of the file at "/home/myuser/supersecret.txt"into the variable section1 -> subsection1.
Similarly, to read a value from an environment variable, add a "_from_env" tothe variable name. For example if we wanted to obtain a value from the variableSPECIFIC_ENV_VAR:
export SPECIFIC_ENV_VAR="some value"
Using a settings file with the contents (in this case yaml):
section1:subsection1_from_env:SPECIFIC_ENV_VAR
or using an environment variable:
export CLIMATECONTROL_SECTION1_SUBSECTION1_FROM_FILE="/home/myuser/supersecret.txt"
will both write "some value" into the variable section1 -> subsection1.
section1_from_json_content:'{"subsection1": "test", "subsection2": 2}'section2_from_toml_content:'subsection1 = "test"\nsubsection2 = 2\n'section3_from_yaml_content:'subsection1: test\nsubsection2: 2\n'
The equivilant environment variables are also handled correctly:
CLIMATECONTROL_SECTION1_FROM_JSON_CONTENT='{"subsection1": "test", "subsection2": 2}'CLIMATECONTROL_SECTION2_FROM_TOML_CONTENT='subsection1 = "test"\nsubsection2 = 2\n'CLIMATECONTROL_SECTION3_FROM_YAML_CONTENT='subsection1: test\nsubsection2: 2\n'
In addition, file variables can also target other settings files directly. Todo this, just make sure the target file is has an extension supported byclimate control. A simple example is illustrated here. Given a settings file:
value1:"spam"section1_from_file:/home/myuser/nestedfile.yaml
where the content of /home/myuser/nestedfile.yaml is:
value2:"cheese"subsection:value3:"parrot"
which would result in a settings structure:
{"value1":"spam","section1": {"value2":"cheese","subsection": {"value3":"parrot" } }}You can also expand the settings at the root of the document by using only"_from_file" as the key:
value1:"spam"_from_file:/home/myuser/nestedfile.yaml
{"value1":"spam","value2":"cheese","subsection": {"value3":"parrot" }}While the default climate object is great for most uses, perhaps you alreadyhave a settings object style that you like or use a specific library forvalidation. In these cases, CLIMATECONTROL can be extended to use theselibraries.
>>>from climatecontrol.ext.dataclassesimport Climate>>>from dataclassesimport dataclass, field>>>>>>@dataclass...classSettingsSubSchema:... d:int=4...>>>@dataclass...classSettingsSchema:... a:str='test'... b:bool=False... c: SettingsSubSchema= field(default_factory=SettingsSubSchema)...>>> climate= Climate(dataclass_cls=SettingsSchema)>>># defaults are initialized automatically:>>> climate.settings.a'test'>>> climate.settings.c.d4>>># Types are checked if given>>> climate.update({'c': {'d':'boom!'}})Traceback (most recent call last): ...dacite.exceptions.WrongTypeError: wrong type for field "c.d" - should be "int" instead of "str"
Pydantic is a great data validation library:https://github.com/samuelcolvin/pydantic and climatecontrol also provides asimple extension to use pydantic models directly (typing functionality mentionedabove works here as well).
>>>from climatecontrol.ext.pydanticimport Climate>>>>>>classSettingsSubSchema(BaseModel):... d:int=4...>>>classSettingsSchema(BaseModel):... a:str='test'... b:bool=False... c: SettingsSubSchema= SettingsSubSchema()...>>> climate= Climate(model=SettingsSchema)>>># defaults are initialized automatically:>>> climate.settings.a'test'>>> climate.settings.c.d4>>># Types are checked if given>>> climate.update({'c': {'d':'boom!'}})Traceback (most recent call last): ...pydantic.error_wrappers.ValidationError: 1 validation error for SettingsSchemac -> d value is not a valid integer (type=type_error.integer)
Theclick library is a great tool for creating command line applications. Ifyou don't want to have to use an environment to set your configuration file.Write your command line application like this:
importclick@click.command()@climate.click_settings_file_option()defcli():print(climate.settings)
save it to a file like "cli.py" and then call it after installing click:
pip install clickpython cli.py --settings ./my_settings_file.toml
whithout needing to set any env vars.
Multiple files are supported. They will be automatically recursively mergedwith the last file overriting any overlapping keys of the first file.
pip install clickpython cli.py --settings ./my_settings_file.toml --settings ./my_settings_file.yaml
If you have a "logging" section in your settings files, you can configurepython standard library logging using that section directly:
logging:formatters:default:format':"%(levelname)s > %(message)s"root:level:DEBUG
importloggingfromclimatecontrolimportclimateclimate.setup_logging()logging.debug('test')# outputs: DEBUG > test
When testing your application, different behaviours often depend on settingstaking on different values. Assuming that you are using a single Settingsobject accross multiple functions or modules, handling these settings changesin tests can be tricky.
The settings object provides a simple method for modifying your settings objecttemporarily:
climate.update({'a':1})# Enter a temporary changes context block:withclimate.temporary_changes():climate.update({'a':1})# Inside the context, the settings can be modified and used as you chooseprint(climate['a'])# outputs: 2# After the context exits the settings mapprint(climate['a'])# outputs: 1
See:CONTRIBUTING.md
About
Python library for loading settings and config data from files and environment variables