LoggingThe Ultimate Guide

your open-source resource for understanding, analyzing, and troubleshooting system logs

curated byloggly

0

Python Logging Libraries and Frameworks

Django

The Django framework is the most popular web application framework for Python. Among the many features it provides is a simple default logging configuration. When the DEBUG setting is True, Django will emit Django log messages to the console (stderr). When DEBUG is False, Django will send its own errors as emails.

These emails are mainly used for internal server errors. They include a backtrace of the error and a lot of extra information about the request useful for debugging. To change who receives the emails from a production Django installation, change the ADMINS setting in settings.py.

Email does not scale well for larger applications, though. To collect and summarize internal error notifications, there are tools with specific Django support, such as Sentry.

For log messages not originating from Django (i.e., log messages on loggers not starting with django.--), programmers will have to set up their own logging as described in the first part of this guide.

One special feature of Django is that it will log all SQL queries it runs under the django.db logger in the level DEBUG. Programmers can use this to watch all SQL queries in the logs by setting the log level to DEBUG. Of course, this can generate a lot of messages, so sometimes it is important to selectively ignore them:

import logging
# Log DEBUG messages and higher in general
log = logging.getLogger()
log.setLevel("DEBUG")

# Except for django.db, stick to INFO level messages and higher there
dblogger = logging.getLogger("django.db")
dblogger.setLevel("INFO")

Flask

Flask is the other widely used web framework for Python. True to the spirit of the library, the logging support is minimal but useful. It simply uses the standard logging library, and programmers can configure the module to write logs where they want them to go as usual. The best practice logging setup to just emit everything to standard output works very well here.

The Flask documentation includes some examples for setting up more elaborate logging, in particular an example on how to mimic Django’s debug emails on internal errors. This shows the flexibility of the logging subsystem, but be aware that problems with the mail server can have side effects on the web application.

Again, though, email does not scale well for larger applications. As with Django, it is easy to submit internal server errors to an external tool like Sentry for summarizing and analysis.

Twisted

The asynchronous programming framework Twisted includes a logging framework completely unrelated to and distinct from the standard logging module in Python. This is mainly to avoid any blocking calls, which would affect the whole Twisted runtime.

As it does not have to be compatible, the framework uses a very different approach, focusing on structured logging. Log messages consist of key-value pairs of Python objects, which then can be serialized into a log file. Twisted recommends a JSON-based log file, but other receivers are possible.

By default, Twisted will emit log messages in a string format to standard output, where it can be picked up by systemd and forwarded to normal system logging. This loses the key-value pairs, though, so Twisted recommends a slightly different approach. Instead of or in addition to sending log messages to standard output, they should be written in a serializing format like JSON to a log file. From there, they can be forwarded to a centralized log system or analyzed directly.

Manually analyzing such structured log files requires some work, though. Twisted does not ship with command line tools to analyze them directly, but instead provides a programming interface that can read in those files again for analysis.

import io
from twisted.logger import eventsFromJSONLogFile

for event in eventsFromJSONLogFile(io.open("log.json")):
   ... analyze event ...

The Twisted documentation on the logging submodule goes into a lot more detail on the topic.

JSON is a structured format that is supported directly by various centralized logging platforms, which makes it much easier to analyze such data. Sadly, Twisted does not support any of them by default.

Written & Contributed by

Jorgen

This guide will help software developers and system administrators become experts at using logs to better run their systems. This is a vendor-neutral, community effort featuring examples from a variety of solutions

Meet Our Contributors Become a contributor