Usage¶

Creating ¶

You can create OmegaConf objects from multiple sources.

Empty ¶

>>>fromomegaconfimportOmegaConf>>>conf=OmegaConf.create()>>>print(OmegaConf.to_yaml(conf)){}

From a dictionary ¶

>>>conf=OmegaConf.create({"k":"v","list":[1,{"a":"1","b":"2",3:"c"}]})>>>print(OmegaConf.to_yaml(conf))k: vlist:- 1- a: '1'  b: '2'  3: c

Here is an example of various supported key types:

>>>fromenumimportEnum>>>classColor(Enum):...RED=1...BLUE=2>>>>>>conf=OmegaConf.create(...{"key":"str",123:"int",True:"bool",3.14:"float",Color.RED:"Color",b"123":"bytes"}...)>>>>>>print(conf){'key': 'str', 123: 'int', True: 'bool', 3.14: 'float', <Color.RED: 1>: 'Color', b'123': 'bytes'}

OmegaConf supportsstr,int,bool,floatbytes, andEnum as dictionary key types.

From a list ¶

>>>conf=OmegaConf.create([1,{"a":10,"b":{"a":10,123:"int_key"}}])>>>print(OmegaConf.to_yaml(conf))- 1- a: 10  b:    a: 10    123: int_key

Tuples are supported as a valid option too.

From a YAML file ¶

>>>conf=OmegaConf.load('source/example.yaml')>>># Output is identical to the YAML file>>>print(OmegaConf.to_yaml(conf))server:  port: 80log:  file: ???  rotation: 3600users:- user1- user2

From a YAML string ¶

>>>s="""...a: b...b: c...list:...- item1...- item2...123: 456...""">>>conf=OmegaConf.create(s)>>>print(OmegaConf.to_yaml(conf))a: bb: clist:- item1- item2123: 456

From a dot-list ¶

>>>dot_list=["a.aa.aaa=1","a.aa.bbb=2","a.bb.aaa=3","a.bb.bbb=4"]>>>conf=OmegaConf.from_dotlist(dot_list)>>>print(OmegaConf.to_yaml(conf))a:  aa:    aaa: 1    bbb: 2  bb:    aaa: 3    bbb: 4

From command line arguments ¶

To parse the content of sys.arg:

>>># Simulating command line arguments>>>sys.argv=['your-program.py','server.port=82','log.file=log2.txt']>>>conf=OmegaConf.from_cli()>>>print(OmegaConf.to_yaml(conf))server:  port: 82log:  file: log2.txt

From structured config ¶

You can create OmegaConf objects from structured config classes or objects. This provides static and runtime type safety.SeeStructured Configs for more details, or keep reading for a minimal example.

>>>fromdataclassesimportdataclass>>>@dataclass...classMyConfig:...port:int=80...host:str="localhost">>># For strict typing purposes, prefer OmegaConf.structured() when creating structured configs>>>conf=OmegaConf.structured(MyConfig)>>>print(OmegaConf.to_yaml(conf))port: 80host: localhost

You can use an object to initialize the config as well:

>>>conf=OmegaConf.structured(MyConfig(port=443))>>>print(OmegaConf.to_yaml(conf))port: 443host: localhost

OmegaConf objects constructed from Structured classes provide runtime type safety:

>>>conf.port=42# Ok, type matches>>>conf.port="1080"# Ok! "1080" can be converted to an int>>>conf.port="oops"# "oops" cannot be converted to an intTraceback (most recent call last):...omegaconf.errors.ValidationError:Value 'oops' could not be converted to Integer

In addition, the config class can be used as type annotation for static type checkers or IDEs:

>>>deffoo(conf:MyConfig):...print(conf.port)# passes static type checker...print(conf.pork)# fails static type checker

Access and manipulation ¶

Input YAML file for this section:

server:port:80log:file:???rotation:3600users:-user1-user2

Access ¶

>>># object style access of dictionary elements>>>conf.server.port80>>># dictionary style access>>>conf['log']['rotation']3600>>># items in list>>>conf.users[0]'user1'

Default values ¶

You can provide default values directly in the accessing code:

>>>conf.get('missing_key','a default value')'a default value'

Mandatory values ¶

Use the value"???" to indicate parameters that need to be set prior to access

>>>conf.log.fileTraceback (most recent call last):...omegaconf.MissingMandatoryValue:log.file

Manipulation ¶

>>># Changing existing keys>>>conf.server.port=81>>># Adding new keys>>>conf.server.hostname="localhost">>># Adding a new dictionary>>>conf.database={'hostname':'database01','port':3306}

Serialization ¶

OmegaConf objects can be saved and loaded with OmegaConf.save() and OmegaConf.load().The created file is in YAML format.Save and load can operate on file-names, Paths and file objects.

Save/Load YAML file ¶

>>>conf=OmegaConf.create({"foo":10,"bar":20,123:456})>>>withtempfile.NamedTemporaryFile()asfp:...OmegaConf.save(config=conf,f=fp.name)...loaded=OmegaConf.load(fp.name)...assertconf==loaded

Note that this does not retain type information.

Save/Load pickle file ¶

Use pickle to save and load while retaining the type information.Note that the saved file may be incompatible across different versions of OmegaConf.

>>>conf=OmegaConf.create({"foo":10,"bar":20,123:456})>>>withtempfile.TemporaryFile()asfp:...pickle.dump(conf,fp)...fp.flush()...assertfp.seek(0)==0...loaded=pickle.load(fp)...assertconf==loaded

Variable interpolation ¶

OmegaConf supports variable interpolation. Interpolations are evaluated lazily on access.

The interpolated variable can be the path to another node in the configuration, and in that casethe value will be the value of that node.This path may use either dot-notation (foo.1), brackets ([foo][1]) or a mix of both (foo[1],[foo].1).

Interpolations are absolute by default. Relative interpolation are prefixed by one or more dots:The first dot denotes the level of the node itself and additional dots are going up the parent hierarchy.e.g.${..foo} points to thefoo sibling of the parent of the current node.

NOTE: Interpolations may cause config cycles. Such cycles are forbidden and may cause undefined behavior.

Input YAML file:

server:  host: localhost  port: 80client:  url: http://${server.host}:${server.port}/  server_port: ${server.port}  # relative interpolation  description: Client of ${.url}

Example:

>>>conf=OmegaConf.load('source/config_interpolation.yaml')>>>defshow(x):...print(f"type:{type(x).__name__}, value:{repr(x)}")>>># Primitive interpolation types are inherited from the reference>>>show(conf.client.server_port)type: int, value: 80>>># String interpolations concatenate fragments into a string>>>show(conf.client.url)type: str, value: 'http://localhost:80/'>>># Relative interpolation example>>>show(conf.client.description)type: str, value: 'Client of http://localhost:80/'

Nested interpolation ¶

Interpolations may be nested, enabling more advanced behavior like dynamically selecting a sub-config:

>>>cfg=OmegaConf.create(...{..."plans":{..."A":"plan A",..."B":"plan B",...},..."selected_plan":"A",..."plan":"${plans[${selected_plan}]}",...}...)>>>cfg.plan# default plan'plan A'>>>cfg.selected_plan="B">>>cfg.plan# new plan'plan B'

Interpolated nodes can be any node in the config, not just leaf nodes:

>>>cfg=OmegaConf.create(...{..."john":{"height":180,"weight":75},..."player":"${john}",...}...)>>>(cfg.player.height,cfg.player.weight)(180, 75)

Resolvers ¶

Add new interpolation types by registering resolvers usingOmegaConf.register_new_resolver().Such resolvers are called when the config node is accessed.The minimal example below shows its most basic usage, seeResolvers for more details.

>>>OmegaConf.register_new_resolver(..."add",lambda*numbers:sum(numbers)...)>>>c=OmegaConf.create({'total':'${add:1,2,3}'})>>>c.total6

Built-in resolvers ¶

OmegaConf comes with a set of built-in custom resolvers:

oc.create: Dynamically generating config nodes
oc.decode: Parsing an input string using interpolation grammar
oc.deprecated: Deprecate a key in your config
oc.env: Accessing environment variables
oc.select: Selecting an interpolation key, similar to interpolation but more flexible
oc.dict.{keys,value}: Viewing the keys or the values of a dictionary as a list

Merging configurations ¶

Merging configurations enables the creation of reusable configuration files for each logical componentinstead of a single config file for each variation of your task.

OmegaConf.merge()¶

Machine learning experiment example:

conf=OmegaConf.merge(base_cfg,model_cfg,optimizer_cfg,dataset_cfg)

Web server configuration example:

conf=OmegaConf.merge(server_cfg,plugin1_cfg,site1_cfg,site2_cfg)

The following example creates two configs from files, and one from the cli. It then combines them into a single object.Note how the port changes to 82.

example2.yaml file:

server:port:80users:-user1-user2

example3.yaml file:

log:file:log.txt

>>>fromomegaconfimportOmegaConf>>>importsys>>>>>># Simulate command line arguments>>>sys.argv=['program.py','server.port=82']>>>>>>base_conf=OmegaConf.load('source/example2.yaml')>>>second_conf=OmegaConf.load('source/example3.yaml')>>>cli_conf=OmegaConf.from_cli()>>>>>># merge them all>>>conf=OmegaConf.merge(base_conf,second_conf,cli_conf)>>>print(OmegaConf.to_yaml(conf))server:  port: 82users:- user1- user2log:  file: log.txt

By default,merge() is replacing the target list with the source list.Uselist_merge_mode to control the merge behavior for lists.This Enum is defined inomegaconf.ListMergeMode and defines the following modes:*REPLACE: Replaces the target list with the new one (default)*EXTEND: Extends the target list with the new one*EXTEND_UNIQUE: Extends the target list items with items not present in it

example2.yaml file:

server:port:80users:-user1-user2

example4.yaml file:

users:-user3-user2

If you load them and merge them withlist_merge_mode=ListMergeMode.EXTEND_UNIQUE you will get this:

>>>fromomegaconfimportOmegaConf,ListMergeMode>>>>>>cfg_1=OmegaConf.load('source/example2.yaml')>>>cfg_2=OmegaConf.load('source/example4.yaml')>>>>>>mode=ListMergeMode.EXTEND_UNIQUE>>>conf=OmegaConf.merge(cfg_1,cfg_2,list_merge_mode=mode)>>>print(OmegaConf.to_yaml(conf))server:  port: 80users:- user1- user2- user3

OmegaConf.unsafe_merge()¶

OmegaConf offers a second faster function to merge config objects:

conf=OmegaConf.unsafe_merge(base_cfg,model_cfg,optimizer_cfg,dataset_cfg)

Unlike OmegaConf.merge(), unsafe_merge() is destroying the input configs and they should no longer be usedafter this call. The upside is that it’s substantially faster.

Configuration flags ¶

OmegaConf support several configuration flags.Configuration flags can be set on any configuration node (Sequence or Mapping). if a configuration flag is not setit inherits the value from the parent of the node.The default value inherited from the root node is always false.

Read-only flag ¶

A read-only configuration cannot be modified.An attempt to modify it will result in omegaconf.ReadonlyConfigError exception

>>>conf=OmegaConf.create({"a":{"b":10}})>>>OmegaConf.set_readonly(conf,True)>>>conf.a.b=20Traceback (most recent call last):...omegaconf.ReadonlyConfigError:a.b

You can temporarily remove the read only flag from a config object:

>>>conf=OmegaConf.create({"a":{"b":10}})>>>OmegaConf.set_readonly(conf,True)>>>withread_write(conf):...conf.a.b=20>>>conf.a.b20

Struct flag ¶

By default, OmegaConf dictionaries allow write access to unknown fields.If a field does not exist, writing it will create the field, and attempting toaccess the field before creation will raise an exception (eitherConfigKeyErrororConfigAttributeError, depending on the mode of access).It’s sometime useful to change this behavior. UsingOmegaConf.set_struct,it is possible to prevent the creation of fields that do not exist:

>>>conf=OmegaConf.create({"a":{"aa":10,"bb":20}})>>>OmegaConf.set_struct(conf,True)>>>conf.a.cc=30Traceback (most recent call last):...omegaconf.errors.ConfigAttributeError:Error setting cc=30 : Key 'cc' is not in struct    full_key: a.cc    reference_type=Any    object_type=dict

You can temporarily remove the struct flag from a config object:

>>>fromomegaconfimportopen_dict>>>conf=OmegaConf.create({"a":{"aa":10,"bb":20}})>>>OmegaConf.set_struct(conf,True)>>>withopen_dict(conf):...conf.a.cc=30>>>conf.a.cc30

Utility functions ¶

OmegaConf.to_container ¶

OmegaConf config objects looks very similar to python dict and list, but in fact are not.UseOmegaConf.to_container(cfg:Container,resolve:bool) to convert to a primitive container.Ifresolve is set toTrue, interpolations will be resolved during conversion.

>>>conf=OmegaConf.create({"foo":"bar","foo2":"${foo}"})>>>asserttype(conf)==DictConfig>>>primitive=OmegaConf.to_container(conf)>>>show(primitive)type: dict, value: {'foo': 'bar', 'foo2': '${foo}'}>>>resolved=OmegaConf.to_container(conf,resolve=True)>>>show(resolved)type: dict, value: {'foo': 'bar', 'foo2': 'bar'}

Using`throw_on_missing`¶

You can control how missing values are handled byOmegaConf.to_container()using thethrow_on_missing keyword argument.

>>>conf=OmegaConf.create({"foo":"bar","missing":"???"})>>>has_missing=OmegaConf.to_container(conf,throw_on_missing=False)>>>show(has_missing)type: dict, value: {'foo': 'bar', 'missing': '???'}>>>OmegaConf.to_container(conf,throw_on_missing=True)Traceback (most recent call last):...omegaconf.errors.MissingMandatoryValue:Missing mandatory value: missing    full_key: missing    object_type=dict

By default,throw_on_missing=False.Settingthrow_on_missing=True can be useful if you want your program tofail fast when there are missing values in the config.

Using`structured_config_mode`¶

You can customize the treatment ofOmegaConf.to_container() forStructured Config nodes using thestructured_config_mode option.The default,structured_config_mode=SCMode.DICT, converts Structured Config nodes to plain dict.

Usingstructured_config_mode=SCMode.DICT_CONFIG causes such nodes to remainasDictConfig, allowing attribute style access on the resulting node.

Usingstructured_config_mode=SCMode.INSTANTIATE, Structured Config nodesare converted to instances of the backing dataclass or attrs class. Note thatwhenstructured_config_mode=SCMode.INSTANTIATE, interpolations nested withina structured config node will be resolved, even ifOmegaConf.to_container is calledwith the the keyword argumentresolve=False, so that interpolations are resolved beforebeing used to instantiate dataclass/attr class instances. Interpolations withinnon-structured parent nodes will be resolved (or not) as usual, according totheresolve keyword arg.

>>>fromomegaconfimportSCMode>>>conf=OmegaConf.create({"structured_config":MyConfig})>>>container=OmegaConf.to_container(conf,...structured_config_mode=SCMode.DICT_CONFIG)>>>show(container)type: dict, value: {'structured_config': {'port': 80, 'host': 'localhost'}}>>>show(container["structured_config"])type: DictConfig, value: {'port': 80, 'host': 'localhost'}

OmegaConf.to_object ¶

TheOmegaConf.to_object method recursively convertsDictConfig andListConfig objectsinto plain Python dicts and lists, with the exception that Structured Config objects areconverted into instances of the backing dataclass or attr class. Interpolations in the config are always resolved byOmegaConf.to_object.

>>>container=OmegaConf.to_object(conf)>>>show(container)type: dict, value: {'structured_config': MyConfig(port=80, host='localhost')}>>>show(container["structured_config"])type: MyConfig, value: MyConfig(port=80, host='localhost')

Note that here,container["structured_config"] is actually an instance ofMyConfig, whereas in the previous examples we had adict or aDictConfig object that was duck-typed to look like an instance ofMyConfig.

The callOmegaConf.to_object(conf) is equivalent toOmegaConf.to_container(conf,resolve=True,throw_on_missing=True,structured_config_mode=SCMode.INSTANTIATE).

OmegaConf.resolve ¶

defresolve(cfg:Container)->None:"""    Resolves all interpolations in the given config object in-place.    :param cfg: An OmegaConf container (DictConfig, ListConfig)                Raises a ValueError if the input object is not an OmegaConf container.    """

Normally interpolations are resolved lazily, at access time.This function eagerly resolves all interpolations in the given config object in-place.Example:

>>>cfg=OmegaConf.create({"a":10,"b":"${a}"})>>>show(cfg)type: DictConfig, value: {'a': 10, 'b': '${a}'}>>>assertcfg.a==cfg.b==10# lazily resolving interpolation>>>OmegaConf.resolve(cfg)>>>show(cfg)type: DictConfig, value: {'a': 10, 'b': 10}

OmegaConf.select ¶

OmegaConf.select() allows you to select a config node or value, using either a dot-notation or brackets to denote sub-keys.

>>>cfg=OmegaConf.create({..."foo":{..."missing":"???",..."bar":{..."zonk":10,...}...}...})>>>assertOmegaConf.select(cfg,"foo")=={..."missing":"???",..."bar":{..."zonk":10,...}...}>>>assertOmegaConf.select(cfg,"foo.bar")=={..."zonk":10,...}>>>assertOmegaConf.select(cfg,"foo.bar.zonk")==10# dots>>>assertOmegaConf.select(cfg,"foo[bar][zonk]")==10# brackets>>>assertOmegaConf.select(cfg,"no_such_key",default=99)==99>>>assertOmegaConf.select(cfg,"foo.missing")isNone>>>assertOmegaConf.select(cfg,"foo.missing",default=99)==99>>>OmegaConf.select(cfg,..."foo.missing",...throw_on_missing=True...)Traceback (most recent call last):...omegaconf.errors.MissingMandatoryValue:missing node selected    full_key: foo.missing

OmegaConf.update ¶

OmegaConf.update() allows you to update values in your config using either a dot-notation or brackets to denote sub-keys.

The merge flag controls the behavior if the input is adict or alist.Ifmerge=True true (the default), dicts and lists are merged instead of being assigned.Theforce_add flag ensures that the path is created even if it will result in insertion of new values into struct nodes.

>>>cfg=OmegaConf.create({"foo":{"bar":10}})>>>OmegaConf.update(cfg,"foo.bar",20)>>>assertcfg.foo.bar==20>>># Set dictionary value (using dot notation)>>>OmegaConf.update(cfg,"foo.bar",{"zonk":30},merge=False)>>>assertcfg.foo.bar=={"zonk":30}>>># Merge dictionary value (using bracket notation)>>># note that merge is True by default, so you don't really need it here.>>>OmegaConf.update(cfg,"foo[bar]",{"oompa":40},merge=True)>>>assertcfg.foo.bar=={"zonk":30,"oompa":40}>>># force_add ignores nodes in struct mode or Structured Configs nodes>>># and updates anyway, inserting keys as needed.>>>OmegaConf.set_struct(cfg,True)>>>OmegaConf.update(cfg,"a.b.c.d",10,force_add=True)>>>assertcfg.a.b.c.d==10

OmegaConf.masked_copy ¶

Creates a copy of aDictConfig that contains only specific keys.

>>>conf=OmegaConf.create({"a":{"b":10},"c":20})>>>print(OmegaConf.to_yaml(conf))a:  b: 10c: 20>>>c=OmegaConf.masked_copy(conf,["a"])>>>print(OmegaConf.to_yaml(c))a:  b: 10

OmegaConf.is_missing ¶

Tests if a value is missing ("???").

>>>cfg=OmegaConf.create({..."foo":10,..."bar":"???"...})>>>assertnotOmegaConf.is_missing(cfg,"foo")>>>assertOmegaConf.is_missing(cfg,"bar")

OmegaConf.is_interpolation ¶

Tests if a value is an interpolation.

>>>cfg=OmegaConf.create({..."foo":10,..."bar":"${foo}"...})>>>assertnotOmegaConf.is_interpolation(cfg,"foo")>>>assertOmegaConf.is_interpolation(cfg,"bar")

OmegaConf.{is_config, is_dict, is_list}¶

OmegaConf.is_config tests whether an object is an OmegaConf object (e.g.DictConfig orListConfig).OmegaConf.is_dict(cfg) is equivalent toisinstance(cfg,DictConfig),andOmegaConf.is_list(cfg) is equivalent toisinstance(cfg,ListConfig).

>>># dict:>>>d=OmegaConf.create({"foo":"bar"})>>>assertOmegaConf.is_config(d)>>>assertOmegaConf.is_dict(d)>>>assertnotOmegaConf.is_list(d)>>># list:>>>l=OmegaConf.create([1,2,3])>>>assertOmegaConf.is_config(l)>>>assertOmegaConf.is_list(l)>>>assertnotOmegaConf.is_dict(l)

OmegaConf.missing_keys ¶

OmegaConf.missing_keys(cfg) returns a set of missing keys present in the inputcfg.Each missing key is represented as astr, using a dotlist style.This utility function can be used after creating a config object, after merging sources and so on,to check for missing mandatory fields and aid in creating a proper error message.

>>>missings=OmegaConf.missing_keys({..."foo":{"bar":"???"},..."missing":"???",..."list":["a",None,"???"]...})>>>assertmissings=={'list[2]','foo.bar','missing'}

The function raises aValueError on input not representing a config.

Debugger integration ¶

OmegaConf is packaged with a PyDev.Debugger extension which enables better debugging experience in PyCharm,VSCode and otherPyDev.Debugger powered IDEs.

The debugger extension enables OmegaConf-aware object inspection:

providing information about interpolations.
properly handling missing values ("???").

The plugin comes in two flavors:

USER: Default behavior, useful when debugging your OmegaConf objects.
DEV: Useful when debugging OmegaConf itself, shows the exact data model of OmegaConf.

The default flavor isUSER. You can select which flavor to use using the environment variableOC_PYDEVD_RESOLVER,Which takes the possible valuesUSER,DEV andDISABLE.

Movatterモバイル変換

Usage¶