LoggingThe Ultimate Guide

your open-source resource for understanding, analyzing, and troubleshooting system logs

curated byloggly

1

Python Logging Basics

Standard Library Logging Module

Python comes with a logging module in the standard library which provides a flexible framework for emitting log messages from Python programs. This module is widely used by libraries and is the first go-to point for most programmers when it comes to logging.

The module provides a way for applications to configure different log handlers and a way of routing log messages to these handlers. This can seem a bit overwhelming at first, but it allows for a highly flexible configuration that can deal with a lot of different use cases.

To emit a log message, a caller first requests a named logger. The name can be used by the application to configure different rules for different loggers. This logger then can be used to emit simply-formatted messages at different log levels (DEBUG, INFO, ERROR, etc.), which again can be used by the application to handle messages of higher priority different than those of a lower priority. While it might sound complicated, it can be as simple as this:

Internally, the message is turned into a LogRecord object and routed to a Handler object registered for this logger. The handler will then use a Formatter to turn the LogRecord into a string and emit that string.

Fortunately programmers do not have to be aware of the details most of the time. The Python documentation contains an excellent article on the logging module and how it all works together. The rest of this article will focus on best practices instead of all the possible ways of using this module.

Logging from Modules

Modules, that is, libraries intended to be used by other programs, should only emit log messages as a best practice. They should never configure how log messages are handled. That is the responsibility of the application that imports and uses the modules. The only responsibility modules have is to make it easy for the application to route their log messages. For this reason, it is a convention for each module to simply use a logger named like the module itself. This makes it easy for the application to route different modules differently, while also keeping log code in the module simple. The module just needs two lines to set up logging, and then use the named logger:

That is all there is to it. In Python, __name__ contains the full name of the current module, so this will simply work in any module.

Configuring Logging

The main application should configure the logging subsystem so log messages go where they should. The Python logging module provides a large number of ways to fine-tune this, but for almost all applications, the configuration can be very simple.

In general, a configuration consists of adding a Formatter and a Handler to the root logger. Because this is so common, the logging module provides a utility function called basicConfig that handles a majority of use cases.

Applications should configure logging as early as possible, preferably as the first thing in the application, so that log messages do not get lost during startup.

Finally, applications should wrap a try/except block around the main application code to send any exceptions through the logging interface instead of just to stderr.

Example 1: Logging to Standard Output for Systemd

This is the simplest and probably the best option for configuring logging these days. When using systemd to run a daemon, applications can just send log messages to stdout or stderr and have systemd forward the messages to journald and syslog. As an additional perk, this does not even require catching exceptions, as Python already writes those to standard error.

That’s it. The application will now log all messages with level INFO or above to stderr, using a simple format:

ERROR:the.module.name:The log message

The application can even be configured to include DEBUG messages, or maybe only ERROR, by setting the LOGLEVEL environment variable.

The only problem with this solution is that exceptions are logged as multiple lines, which can cause problems for later analysis. Sadly, configuring Python to send multi-line exceptions as a single line is not quite as simple, but certainly possible.

Example 2: Syslog

The alternative is to send it directly to syslog. This is great for older operating systems that don’t have systemd. In an ideal world, this should be simple, but sadly Python requires a bit more elaborate configuration to be able to send unicode log messages.

Example 3: Log File

The final option is to log messages directly to a file. This is rarely useful these days, as administrators can configure syslog to write certain messages to specific files, and if they use centralized logging, having to deal with additional log files is annoying. But it is an option.

When logging to files, the main thing to beware of is that log files need to be rotated regularly. The application needs to detect the log file being renamed and handle that situation. While Python provides its own file rotation handler, it is best to leave log rotation to dedicated tools such as logrotate. The WatchedFileHandler will keep track of the log file and reopen it if it is rotated, making it work well with logrotate without requiring any specific signals.

Other Destinations

It is possible to use other log destinations, and certain frameworks make good use of this (e.g., Django can send certain log messages as email). The HTTPHandler might be useful when you are running in a PaaS and you don’t have direct host access to set up syslog or are behind a firewall that blocks outbound syslog, and can be used to log directly to centralized logging systems like Loggly.

Many of the more elaborate log handlers in the logging library can easily block the application, causing outages simply because the logging infrastructure was unresponsive. For these reasons, it is best to keep the logging configuration of an application as simple as possible.

If fine-grained configuration is desirable, the logging module also provides the ability to load the logging configuration from a configuration file. This is very powerful, but rarely necessary. When loading logging configuration from a file, make sure to specify disable_existing_loggers=False. The default, which is there for backwards compatibility only, will disable any loggers created by modules. This breaks many modules.

Summary

Logging in Python is simple and well standardized, thanks to a powerful logging framework right in the standard library.

Modules should simply log everything to a logger instance for their module name. This makes it easy for the application to route log messages of different modules to different places, if necessary.

Applications then have a lot of options to configure logging. In a modern infrastructure, though, following best practices simplifies this a lot. Unless specifically needed, simply logging to stdout/stderr and letting systemd handle log messages is sufficient and the best approach.

  • Alessandro Pocaterra

    Thank you, well explained.

Written & Contributed by

Jorgen

This guide will help software developers and system administrators become experts at using logs to better run their systems. This is a vendor-neutral, community effort featuring examples from a variety of solutions

Meet Our Contributors Become a contributor