Object connections

The schema is intended to describe a set of logging objects - loggers,handlers, formatters, filters - which are connected to each other inan object graph. Thus, the schema needs to represent connectionsbetween the objects. For example, say that, once configured, aparticular logger has attached to it a particular handler. For thepurposes of this discussion, we can say that the logger represents thesource, and the handler the destination, of a connection between thetwo. Of course in the configured objects this is represented by thelogger holding a reference to the handler. In the configuration dict,this is done by giving each destination object an id which identifiesit unambiguously, and then using the id in the source object’sconfiguration to indicate that a connection exists between the sourceand the destination object with that id.

So, for example, consider the following YAML snippet:

formatters:brief:# configuration for formatter with id 'brief' goes hereprecise:# configuration for formatter with id 'precise' goes herehandlers:h1:#This is an id# configuration of handler with id 'h1' goes hereformatter:briefh2:#This is another id# configuration of handler with id 'h2' goes hereformatter:preciseloggers:foo.bar.baz:# other configuration for logger 'foo.bar.baz'handlers:[h1,h2]

(Note: YAML will be used in this document as it is a little morereadable than the equivalent Python source form for the dictionary.)

The ids for loggers are the logger names which would be usedprogrammatically to obtain a reference to those loggers, e.g.foo.bar.baz. The ids for Formatters and Filters can be any stringvalue (such asbrief,precise above) and they are transient,in that they are only meaningful for processing the configurationdictionary and used to determine connections between objects, and arenot persisted anywhere when the configuration call is complete.

Handler ids are treated specially, see the section onHandler Ids, below.

The above snippet indicates that logger namedfoo.bar.baz shouldhave two handlers attached to it, which are described by the handleridsh1 andh2. The formatter forh1 is that described by idbrief, and the formatter forh2 is that described by idprecise.

User-defined objects

The schema should support user-defined objects for handlers, filtersand formatters. (Loggers do not need to have different types fordifferent instances, so there is no support - in the configuration -for user-defined logger classes.)

Objects to be configured will typically be described by dictionarieswhich detail their configuration. In some places, the logging systemwill be able to infer from the context how an object is to beinstantiated, but when a user-defined object is to be instantiated,the system will not know how to do this. In order to provide completeflexibility for user-defined object instantiation, the user will needto provide a ‘factory’ - a callable which is called with aconfiguration dictionary and which returns the instantiated object.This will be signalled by an absolute import path to the factory beingmade available under the special key'()'. Here’s a concreteexample:

formatters:brief:format:'%(message)s'default:format:'%(asctime)s%(levelname)-8s%(name)-15s%(message)s'datefmt:'%Y-%m-%d %H:%M:%S'custom:():my.package.customFormatterFactorybar:bazspam:99.9answer:42

The above YAML snippet defines three formatters. The first, with idbrief, is a standardlogging.Formatter instance with thespecified format string. The second, with iddefault, has alonger format and also defines the time format explicitly, and willresult in alogging.Formatter initialized with those two formatstrings. Shown in Python source form, thebrief anddefaultformatters have configuration sub-dictionaries:

{'format':'%(message)s'}

and:

{'format':'%(asctime)s%(levelname)-8s%(name)-15s%(message)s','datefmt':'%Y-%m-%d %H:%M:%S'}

respectively, and as these dictionaries do not contain the special key'()', the instantiation is inferred from the context: as a result,standardlogging.Formatter instances are created. Theconfiguration sub-dictionary for the third formatter, with idcustom, is:

{'()':'my.package.customFormatterFactory','bar':'baz','spam':99.9,'answer':42}

and this contains the special key'()', which means thatuser-defined instantiation is wanted. In this case, the specifiedfactory callable will be used. If it is an actual callable it will beused directly - otherwise, if you specify a string (as in the example)the actual callable will be located using normal import mechanisms.The callable will be called with theremaining items in theconfiguration sub-dictionary as keyword arguments. In the aboveexample, the formatter with idcustom will be assumed to bereturned by the call:

my.package.customFormatterFactory(bar='baz',spam=99.9,answer=42)

The key'()' has been used as the special key because it is not avalid keyword parameter name, and so will not clash with the names ofthe keyword arguments used in the call. The'()' also serves as amnemonic that the corresponding value is a callable.

Access to external objects

There are times where a configuration will need to refer to objectsexternal to the configuration, for examplesys.stderr. If theconfiguration dict is constructed using Python code then this isstraightforward, but a problem arises when the configuration isprovided via a text file (e.g. JSON, YAML). In a text file, there isno standard way to distinguishsys.stderr from the literal string'sys.stderr'. To facilitate this distinction, the configurationsystem will look for certain special prefixes in string values andtreat them specially. For example, if the literal string'ext://sys.stderr' is provided as a value in the configuration,then theext:// will be stripped off and the remainder of thevalue processed using normal import mechanisms.

The handling of such prefixes will be done in a way analogous toprotocol handling: there will be a generic mechanism to look forprefixes which match the regular expression^(?P<prefix>[a-z]+)://(?P<suffix>.*)$ whereby, if theprefixis recognised, thesuffix is processed in a prefix-dependentmanner and the result of the processing replaces the string value. Ifthe prefix is not recognised, then the string value will be leftas-is.

The implementation will provide for a set of standard prefixes such asext:// but it will be possible to disable the mechanism completelyor provide additional or different prefixes for special handling.

Access to internal objects

As well as external objects, there is sometimes also a need to referto objects in the configuration. This will be done implicitly by theconfiguration system for things that it knows about. For example, thestring value'DEBUG' for alevel in a logger or handler willautomatically be converted to the valuelogging.DEBUG, and thehandlers,filters andformatter entries will take anobject id and resolve to the appropriate destination object.

However, a more generic mechanism needs to be provided for the caseof user-defined objects which are not known to logging. For example,take the instance oflogging.handlers.MemoryHandler, which takesatarget which is another handler to delegate to. Since the systemalready knows about this class, then in the configuration, the giventarget just needs to be the object id of the relevant targethandler, and the system will resolve to the handler from the id. If,however, a user defines amy.package.MyHandler which has aalternate handler, the configuration system would not know thatthealternate referred to a handler. To cater for this, ageneric resolution system will be provided which allows the user tospecify:

handlers:file:# configuration of file handler goes herecustom:():my.package.MyHandleralternate:cfg://handlers.file

The literal string'cfg://handlers.file' will be resolved in ananalogous way to the strings with theext:// prefix, but lookingin the configuration itself rather than the import namespace. Themechanism will allow access by dot or by index, in a similar way tothat provided bystr.format. Thus, given the following snippet:

handlers:email:class:logging.handlers.SMTPHandlermailhost:localhostfromaddr:my_app@domain.tldtoaddrs:-support_team@domain.tld-dev_team@domain.tldsubject:Houston,wehaveaproblem.

in the configuration, the string'cfg://handlers' would resolve tothe dict with keyhandlers, the string'cfg://handlers.emailwould resolve to the dict with keyemail in thehandlers dict,and so on. The string'cfg://handlers.email.toaddrs[1] wouldresolve to'dev_team.domain.tld' and the string'cfg://handlers.email.toaddrs[0]' would resolve to the value'support_team@domain.tld'. Thesubject value could be accessedusing either'cfg://handlers.email.subject' or, equivalently,'cfg://handlers.email[subject]'. The latter form only needs to beused if the key contains spaces or non-alphanumeric characters. If anindex value consists only of decimal digits, access will be attemptedusing the corresponding integer value, falling back to the stringvalue if needed.

Given a stringcfg://handlers.myhandler.mykey.123, this willresolve toconfig_dict['handlers']['myhandler']['mykey']['123'].If the string is specified ascfg://handlers.myhandler.mykey[123],the system will attempt to retrieve the value fromconfig_dict['handlers']['myhandler']['mykey'][123], and fall backtoconfig_dict['handlers']['myhandler']['mykey']['123'] if thatfails.

Handler Ids

Some specific logging configurations require the use of handler levelsto achieve the desired effect. However, unlike loggers which canalways be identified by their names, handlers have no persistenthandles whereby levels can be changed via an incremental configurationcall.

Therefore, this PEP proposes to add an optionalname property tohandlers. If used, this will add an entry in a dictionary which mapsthe name to the handler. (The entry will be removed when the handleris closed.) When an incremental configuration call is made, handlerswill be looked up in this dictionary to set the handler levelaccording to the value in the configuration. See the section onincremental configuration for more details.

In theory, such a “persistent name” facility could also be providedfor Filters and Formatters. However, there is not a strong case to bemade for being able to configure these incrementally. On the basisthat practicality beats purity, only Handlers will be given this newname property. The id of a handler in the configuration willbecome itsname.

The handler name lookup dictionary is for configuration use only andwill not become part of the public API for the package.