When to use logging¶

You can access logging functionality by creating a logger vialogger=getLogger(__name__), and then calling the logger’sdebug(),info(),warning(),error() andcritical() methods. To determine when to use logging, and to seewhich logger methods to use when, see the table below. It states, for each of aset of common tasks, the best tool to use for that task.

The logger methods are named after the level or severity of the eventsthey are used to track. The standard levels and their applicability aredescribed below (in increasing order of severity):

The default level isWARNING, which means that only events of this severity and higherwill be tracked, unless the logging package is configured to do otherwise.

Events that are tracked can be handled in different ways. The simplest way ofhandling tracked events is to print them to the console. Another common wayis to write them to a disk file.

A simple example¶

A very simple example is:

importlogginglogging.warning('Watch out!')# will print a message to the consolelogging.info('I told you so')# will not print anything

If you type these lines into a script and run it, you’ll see:

WARNING:root:Watch out!

printed out on the console. TheINFO message doesn’t appear because thedefault level isWARNING. The printed message includes the indication of thelevel and the description of the event provided in the logging call, i.e.‘Watch out!’. The actual output can be formatted quite flexibly if you needthat; formatting options will also be explained later.

Notice that in this example, we use functions directly on theloggingmodule, likelogging.debug, rather than creating a logger and callingfunctions on it. These functions operation on the root logger, but can be usefulas they will callbasicConfig() for you if it has not been called yet, like inthis example. In larger programs you’ll usually want to control the loggingconfiguration explicitly however - so for that reason as well as others, it’sbetter to create loggers and call their methods.

Logging to a file¶

A very common situation is that of recording logging events in a file, so let’slook at that next. Be sure to try the following in a newly started Pythoninterpreter, and don’t just continue from the session described above:

importlogginglogger=logging.getLogger(__name__)logging.basicConfig(filename='example.log',encoding='utf-8',level=logging.DEBUG)logger.debug('This message should go to the log file')logger.info('So should this')logger.warning('And this, too')logger.error('And non-ASCII stuff, too, like Øresund and Malmö')

And now if we open the file and look at what we have, we should find the logmessages:

DEBUG:__main__:This message should go to the log fileINFO:__main__:So should thisWARNING:__main__:And this, tooERROR:__main__:And non-ASCII stuff, too, like Øresund and Malmö

This example also shows how you can set the logging level which acts as thethreshold for tracking. In this case, because we set the threshold toDEBUG, all of the messages were printed.

If you want to set the logging level from a command-line option such as:

--log=INFO

and you have the value of the parameter passed for--log in some variableloglevel, you can use:

getattr(logging,loglevel.upper())

to get the value which you’ll pass tobasicConfig() via thelevelargument. You may want to error check any user input value, perhaps as in thefollowing example:

# assuming loglevel is bound to the string value obtained from the# command line argument. Convert to upper case to allow the user to# specify --log=DEBUG or --log=debugnumeric_level=getattr(logging,loglevel.upper(),None)ifnotisinstance(numeric_level,int):raiseValueError('Invalid log level:%s'%loglevel)logging.basicConfig(level=numeric_level,...)

The call tobasicConfig() should comebefore any calls to a logger’smethods such asdebug(),info(), etc. Otherwise,that logging event may not be handled in the desired manner.

If you run the above script several times, the messages from successive runsare appended to the fileexample.log. If you want each run to start afresh,not remembering the messages from earlier runs, you can specify thefilemodeargument, by changing the call in the above example to:

logging.basicConfig(filename='example.log',filemode='w',level=logging.DEBUG)

The output will be the same as before, but the log file is no longer appendedto, so the messages from earlier runs are lost.

Logging variable data¶

To log variable data, use a format string for the event description message andappend the variable data as arguments. For example:

importlogginglogging.warning('%s before you%s','Look','leap!')

will display:

WARNING:root:Look before you leap!

As you can see, merging of variable data into the event description messageuses the old, %-style of string formatting. This is for backwardscompatibility: the logging package pre-dates newer formatting options such asstr.format() andstring.Template. These newer formattingoptionsare supported, but exploring them is outside the scope of thistutorial: seeUsing particular formatting styles throughout your application for more information.

Changing the format of displayed messages¶

To change the format which is used to display messages, you need tospecify the format you want to use:

importlogginglogging.basicConfig(format='%(levelname)s:%(message)s',level=logging.DEBUG)logging.debug('This message should appear on the console')logging.info('So should this')logging.warning('And this, too')

which would print:

DEBUG:This message should appear on the consoleINFO:So should thisWARNING:And this, too

Notice that the ‘root’ which appeared in earlier examples has disappeared. Fora full set of things that can appear in format strings, you can refer to thedocumentation forLogRecord attributes, but for simple usage, you justneed thelevelname (severity),message (event description, includingvariable data) and perhaps to display when the event occurred. This isdescribed in the next section.

Displaying the date/time in messages¶

To display the date and time of an event, you would place ‘%(asctime)s’ inyour format string:

importlogginglogging.basicConfig(format='%(asctime)s%(message)s')logging.warning('is when this event was logged.')

which should print something like this:

2010-12-12 11:41:42,612 is when this event was logged.

The default format for date/time display (shown above) is like ISO8601 orRFC 3339. If you need more control over the formatting of the date/time, provideadatefmt argument tobasicConfig, as in this example:

importlogginglogging.basicConfig(format='%(asctime)s%(message)s',datefmt='%m/%d/%Y %I:%M:%S %p')logging.warning('is when this event was logged.')

which would display something like this:

12/12/2010 11:46:36 AM is when this event was logged.

The format of thedatefmt argument is the same as supported bytime.strftime().

Next Steps¶

That concludes the basic tutorial. It should be enough to get you up andrunning with logging. There’s a lot more that the logging package offers, butto get the best out of it, you’ll need to invest a little more of your time inreading the following sections. If you’re ready for that, grab some of yourfavourite beverage and carry on.

If your logging needs are simple, then use the above examples to incorporatelogging into your own scripts, and if you run into problems or don’tunderstand something, please post a question on the comp.lang.python Usenetgroup (available athttps://groups.google.com/g/comp.lang.python) and youshould receive help before too long.

Still here? You can carry on reading the next few sections, which provide aslightly more advanced/in-depth tutorial than the basic one above. After that,you can take a look at theLogging Cookbook.

Logging Flow¶

The flow of log event information in loggers and handlers is illustrated in thefollowing diagram.

Loggers¶

Logger objects have a threefold job. First, they expose severalmethods to application code so that applications can log messages at runtime.Second, logger objects determine which log messages to act upon based uponseverity (the default filtering facility) or filter objects. Third, loggerobjects pass along relevant log messages to all interested log handlers.

The most widely used methods on logger objects fall into two categories:configuration and message sending.

These are the most common configuration methods:

• Logger.setLevel() specifies the lowest-severity log message a loggerwill handle, where debug is the lowest built-in severity level and criticalis the highest built-in severity. For example, if the severity level isINFO, the logger will handle only INFO, WARNING, ERROR, and CRITICAL messagesand will ignore DEBUG messages.

• Logger.addHandler() andLogger.removeHandler() add and removehandler objects from the logger object. Handlers are covered in more detailinHandlers.

• Logger.addFilter() andLogger.removeFilter() add and remove filterobjects from the logger object. Filters are covered in more detail inFilter Objects.

You don’t need to always call these methods on every logger you create. See thelast two paragraphs in this section.

With the logger object configured, the following methods create log messages:

• Logger.debug(),Logger.info(),Logger.warning(),Logger.error(), andLogger.critical() all create log records witha message and a level that corresponds to their respective method names. Themessage is actually a format string, which may contain the standard stringsubstitution syntax of%s,%d,%f, and so on. Therest of their arguments is a list of objects that correspond with thesubstitution fields in the message. With regard to**kwargs, thelogging methods care only about a keyword ofexc_info and use it todetermine whether to log exception information.

• Logger.exception() creates a log message similar toLogger.error(). The difference is thatLogger.exception() dumps astack trace along with it. Call this method only from an exception handler.

• Logger.log() takes a log level as an explicit argument. This is alittle more verbose for logging messages than using the log level conveniencemethods listed above, but this is how to log at custom log levels.

getLogger() returns a reference to a logger instance with the specifiedname if it is provided, orroot if not. The names are period-separatedhierarchical structures. Multiple calls togetLogger() with the same namewill return a reference to the same logger object. Loggers that are furtherdown in the hierarchical list are children of loggers higher up in the list.For example, given a logger with a name offoo, loggers with names offoo.bar,foo.bar.baz, andfoo.bam are all descendants offoo.

Loggers have a concept ofeffective level. If a level is not explicitly seton a logger, the level of its parent is used instead as its effective level.If the parent has no explicit level set,its parent is examined, and so on -all ancestors are searched until an explicitly set level is found. The rootlogger always has an explicit level set (WARNING by default). When decidingwhether to process an event, the effective level of the logger is used todetermine whether the event is passed to the logger’s handlers.

Child loggers propagate messages up to the handlers associated with theirancestor loggers. Because of this, it is unnecessary to define and configurehandlers for all the loggers an application uses. It is sufficient toconfigure handlers for a top-level logger and create child loggers as needed.(You can, however, turn off propagation by setting thepropagateattribute of a logger toFalse.)

Handlers¶

Handler objects are responsible for dispatching theappropriate log messages (based on the log messages’ severity) to the handler’sspecified destination.Logger objects can add zero or more handlerobjects to themselves with anaddHandler() method. As an examplescenario, an application may want to send all log messages to a log file, alllog messages of error or higher to stdout, and all messages of critical to anemail address. This scenario requires three individual handlers where eachhandler is responsible for sending messages of a specific severity to a specificlocation.

The standard library includes quite a few handler types (seeUseful Handlers); the tutorials use mainlyStreamHandler andFileHandler in its examples.

There are very few methods in a handler for application developers to concernthemselves with. The only handler methods that seem relevant for applicationdevelopers who are using the built-in handler objects (that is, not creatingcustom handlers) are the following configuration methods:

• ThesetLevel() method, just as in logger objects, specifies thelowest severity that will be dispatched to the appropriate destination. Whyare there twosetLevel() methods? The level set in the loggerdetermines which severity of messages it will pass to its handlers. The levelset in each handler determines which messages that handler will send on.

• setFormatter() selects a Formatter object for this handler touse.

• addFilter() andremoveFilter() respectivelyconfigure and deconfigure filter objects on handlers.

Application code should not directly instantiate and use instances ofHandler. Instead, theHandler class is a base class thatdefines the interface that all handlers should have and establishes somedefault behavior that child classes can use (or override).

Formatters¶

Formatter objects configure the final order, structure, and contents of the logmessage. Unlike the baselogging.Handler class, application code mayinstantiate formatter classes, although you could likely subclass the formatterif your application needs special behavior. The constructor takes threeoptional arguments – a message format string, a date format string and a styleindicator.

logging.Formatter.__init__(fmt=None,datefmt=None,style='%')¶

If there is no message format string, the default is to use theraw message. If there is no date format string, the default date format is:

%Y-%m-%d %H:%M:%S

with the milliseconds tacked on at the end. Thestyle is one of'%','{', or'$'. If one of these is not specified, then'%' will be used.

If thestyle is'%', the message format string uses%(<dictionarykey>)s styled string substitution; the possible keys aredocumented inLogRecord attributes. If the style is'{', the messageformat string is assumed to be compatible withstr.format() (usingkeyword arguments), while if the style is'$' then the message format stringshould conform to what is expected bystring.Template.substitute().

The following message format string will log the time in a human-readableformat, the severity of the message, and the contents of the message, in thatorder:

'%(asctime)s -%(levelname)s -%(message)s'

Formatters use a user-configurable function to convert the creation time of arecord to a tuple. By default,time.localtime() is used; to change thisfor a particular formatter instance, set theconverter attribute of theinstance to a function with the same signature astime.localtime() ortime.gmtime(). To change it for all formatters, for example if you wantall logging times to be shown in GMT, set theconverter attribute in theFormatter class (totime.gmtime for GMT display).

Configuring Logging¶

Programmers can configure logging in three ways:

1. Creating loggers, handlers, and formatters explicitly using Pythoncode that calls the configuration methods listed above.

2. Creating a logging config file and reading it using thefileConfig()function.

3. Creating a dictionary of configuration information and passing itto thedictConfig() function.

For the reference documentation on the last two options, seeConfiguration functions. The following example configures a very simplelogger, a console handler, and a simple formatter using Python code:

importlogging# create loggerlogger=logging.getLogger('simple_example')logger.setLevel(logging.DEBUG)# create console handler and set level to debugch=logging.StreamHandler()ch.setLevel(logging.DEBUG)# create formatterformatter=logging.Formatter('%(asctime)s -%(name)s -%(levelname)s -%(message)s')# add formatter to chch.setFormatter(formatter)# add ch to loggerlogger.addHandler(ch)# 'application' codelogger.debug('debug message')logger.info('info message')logger.warning('warn message')logger.error('error message')logger.critical('critical message')

Running this module from the command line produces the following output:

$pythonsimple_logging_module.py2005-03-19 15:10:26,618 - simple_example - DEBUG - debug message2005-03-19 15:10:26,620 - simple_example - INFO - info message2005-03-19 15:10:26,695 - simple_example - WARNING - warn message2005-03-19 15:10:26,697 - simple_example - ERROR - error message2005-03-19 15:10:26,773 - simple_example - CRITICAL - critical message

The following Python module creates a logger, handler, and formatter nearlyidentical to those in the example listed above, with the only difference beingthe names of the objects:

importloggingimportlogging.configlogging.config.fileConfig('logging.conf')# create loggerlogger=logging.getLogger('simpleExample')# 'application' codelogger.debug('debug message')logger.info('info message')logger.warning('warn message')logger.error('error message')logger.critical('critical message')

Here is the logging.conf file:

[loggers]keys=root,simpleExample[handlers]keys=consoleHandler[formatters]keys=simpleFormatter[logger_root]level=DEBUGhandlers=consoleHandler[logger_simpleExample]level=DEBUGhandlers=consoleHandlerqualname=simpleExamplepropagate=0[handler_consoleHandler]class=StreamHandlerlevel=DEBUGformatter=simpleFormatterargs=(sys.stdout,)[formatter_simpleFormatter]format=%(asctime)s - %(name)s - %(levelname)s - %(message)s

The output is nearly identical to that of the non-config-file-based example:

$pythonsimple_logging_config.py2005-03-19 15:38:55,977 - simpleExample - DEBUG - debug message2005-03-19 15:38:55,979 - simpleExample - INFO - info message2005-03-19 15:38:56,054 - simpleExample - WARNING - warn message2005-03-19 15:38:56,055 - simpleExample - ERROR - error message2005-03-19 15:38:56,130 - simpleExample - CRITICAL - critical message

You can see that the config file approach has a few advantages over the Pythoncode approach, mainly separation of configuration and code and the ability ofnoncoders to easily modify the logging properties.

Note that the class names referenced in config files need to be either relativeto the logging module, or absolute values which can be resolved using normalimport mechanisms. Thus, you could use eitherWatchedFileHandler (relative to the logging module) ormypackage.mymodule.MyHandler (for a class defined in packagemypackageand modulemymodule, wheremypackage is available on the Python importpath).

In Python 3.2, a new means of configuring logging has been introduced, usingdictionaries to hold configuration information. This provides a superset of thefunctionality of the config-file-based approach outlined above, and is therecommended configuration method for new applications and deployments. Becausea Python dictionary is used to hold configuration information, and since youcan populate that dictionary using different means, you have more options forconfiguration. For example, you can use a configuration file in JSON format,or, if you have access to YAML processing functionality, a file in YAMLformat, to populate the configuration dictionary. Or, of course, you canconstruct the dictionary in Python code, receive it in pickled form over asocket, or use whatever approach makes sense for your application.

Here’s an example of the same configuration as above, in YAML format forthe new dictionary-based approach:

version:1formatters:simple:format:'%(asctime)s-%(name)s-%(levelname)s-%(message)s'handlers:console:class:logging.StreamHandlerlevel:DEBUGformatter:simplestream:ext://sys.stdoutloggers:simpleExample:level:DEBUGhandlers:[console]propagate:noroot:level:DEBUGhandlers:[console]

For more information about logging using a dictionary, seeConfiguration functions.

What happens if no configuration is provided¶

If no logging configuration is provided, it is possible to have a situationwhere a logging event needs to be output, but no handlers can be found tooutput the event.

The event is output using a ‘handler of last resort’, stored inlastResort. This internal handler is not associated with anylogger, and acts like aStreamHandler which writes theevent description message to the current value ofsys.stderr (thereforerespecting any redirections which may be in effect). No formatting isdone on the message - just the bare event description message is printed.The handler’s level is set toWARNING, so all events at this andgreater severities will be output.

Configuring Logging for a Library¶

When developing a library which uses logging, you should take care todocument how the library uses logging - for example, the names of loggersused. Some consideration also needs to be given to its logging configuration.If the using application does not use logging, and library code makes loggingcalls, then (as described in the previous section) events of severityWARNING and greater will be printed tosys.stderr. This is regarded asthe best default behaviour.

If for some reason youdon’t want these messages printed in the absence ofany logging configuration, you can attach a do-nothing handler to the top-levellogger for your library. This avoids the message being printed, since a handlerwill always be found for the library’s events: it just doesn’t produce anyoutput. If the library user configures logging for application use, presumablythat configuration will add some handlers, and if levels are suitablyconfigured then logging calls made in library code will send output to thosehandlers, as normal.

A do-nothing handler is included in the logging package:NullHandler (since Python 3.1). An instance of this handlercould be added to the top-level logger of the logging namespace used by thelibrary (if you want to prevent your library’s logged events being output tosys.stderr in the absence of logging configuration). If all logging by alibraryfoo is done using loggers with names matching ‘foo.x’, ‘foo.x.y’,etc. then the code:

importlogginglogging.getLogger('foo').addHandler(logging.NullHandler())

should have the desired effect. If an organisation produces a number oflibraries, then the logger name specified can be ‘orgname.foo’ rather thanjust ‘foo’.

Custom Levels¶

Defining your own levels is possible, but should not be necessary, as theexisting levels have been chosen on the basis of practical experience.However, if you are convinced that you need custom levels, great care shouldbe exercised when doing this, and it is possiblya very bad idea to definecustom levels if you are developing a library. That’s because if multiplelibrary authors all define their own custom levels, there is a chance thatthe logging output from such multiple libraries used together will bedifficult for the using developer to control and/or interpret, because agiven numeric value might mean different things for different libraries.