Ultimate Guide to Logging

Your open-source resource for understanding, analyzing, and troubleshooting system logs

Python Logging Libraries and Frameworks

Django

The Django framework is the most popular web application framework for Python. Among the many features it provides, simple default logging configuration is one of them. When the DEBUG setting is True, Django will emit log messages to the console (stderr). When DEBUG is False, Django will send its own errors as emails.

These emails are mainly used for internal server errors. They include a back trace of the error and a ton of extra information about the request that is useful for debugging. To change who receives the emails from a production Django installation, change the ADMINS setting in settings.py.

Email does not scale well for larger applications, though. To collect and summarize internal error notifications, there are tools with specific Django support, such as Sentry.

For log messages not originating from Django (i.e., log messages on loggers not starting with django.--), programmers will have to set up their own logging as described in the first part of this guide.

One special feature of Django is that it will log all SQL queries it runs under the django.db logger in the level DEBUG. Programmers can use this to watch all SQL queries in the logs by setting the log level to DEBUG. Of course, this can generate many messages, so sometimes it is important to selectively ignore them:

import logging
# Log DEBUG messages and higher in general
log = logging.getLogger()
log.setLevel("DEBUG")

# Except for django.db, stick to INFO level messages and higher there
dblogger = logging.getLogger("django.db")
dblogger.setLevel("INFO")

Flask

Flask is the other widely used web framework for Python. True to the spirit of the library, the logging support is minimal but useful. It simply uses the standard logging library, and programmers can configure the module to write logs where they want them to go, as usual. The best practice logging setup to just emit everything to standard output works very well here.

The Flask documentation includes some examples for setting up more elaborate logging, in particular an example on how to mimic the Django debug emails on internal errors. This shows the flexibility of the logging subsystem, but be aware that problems with the mail server can have side effects on the web application.

Again, though, email does not scale well for larger applications. As with Django, it is easy to submit internal server errors to an external tool like Sentry for summarizing and analysis.

Twisted

The asynchronous programming framework Twisted includes a logging framework completely unrelated to and distinct from the standard logging module in Python. This is mainly to avoid any blocking calls, which would affect the whole Twisted runtime.

As it does not have to be compatible, the framework uses a very different approach, focusing on structured logging. Log messages consist of key-value pairs of Python objects, which then can be serialized into a log file. Twisted recommends a JSON-based log file, but other receivers are possible.

By default, Twisted will emit log messages in a string format to standard output, where it can be picked up by systemd and forwarded to normal system logging. This loses the key-value pairs, though, so Twisted recommends a slightly different approach. Instead of, or in addition to, sending log messages to standard output, they should be written in a serializing format like JSON to a log file. From there, they can be forwarded to a centralized log system or analyzed directly.

Manually analyzing such structured log files requires some work, though. Twisted does not ship with command line tools to analyze them directly, but instead provides a programming interface that can read in those files again for analysis.

import io
from twisted.logger import eventsFromJSONLogFile

for event in eventsFromJSONLogFile(io.open("log.json")):
   ... analyze event ...

The Twisted documentation on the logging submodule goes into a lot more detail on the topic.

JSON is a structured format that is supported directly by various centralized logging platforms, which makes it much easier to analyze such data. Sadly, Twisted does not support any of them by default.