Logging is critical if you want to diagnose problems or understand what’s happening with your Python application. In this post, I hope to convince you that using Python’s logging library is preferable to just putting ‘print’ statements everywhere in your code.

The Python logging library lets you:

• Control what’s emitted
• Define what types of information you want to include in your logs
• Configure how it looks when it’s emitted
• Most importantly, set the destination for your logs

First, Some Python Logging Basics

Here’s a bare-bones example of logging in Python:

You import the logging library, you define here where you want log events to go, and then you emit a log event.

The Python Library Can Add Context

With the Python library at your service, you can include several more types of metadata in your logs. By doing so, you’ll gain valuable context around your log messages to help you diagnose a problem or figure out what’s going on in your system.

You Can Log Severity Levels

Segmenting log events by severity level is a good way to sift through which log messages may be most relevant at a given time. A log event’s severity level also gives you an indication of how worried you should be when you see a particular message.

When you’re still developing your code, debug information is very useful. You want to have the most detailed diagnostic log messages available.

When you’re running your application, log info messages. These are just things that say, “I opened a socket” or “I got a request” or “The database record is inserted correctly,” for example.

Warnings are things that aren’t quite problems yet but might be signs of a future problem. For example, “The disk is 80% full” or “Latency is starting to creep up.” Warning messages tell you what to keep an eye on so that you can take care of business before a problem erupts.

Errors are things that demand immediate attention. If you see a message like, “I tried to reconnect, but the database isn’t there,” you should be worried. Now.

And finally, criticals are indications of big problems. Your program is not likely to be able to continue, and users won’t be able to use the application.

Here’s a quick example of how you can set your logging level in your logger to log only things that are error and above. So, the warning message “Citizens of Earth, be warned” is not displayed. However, the critical message “Citizens of Earth, your planet will be removed NOW” does appear in the console when you run your program.

 

You Can Use Timestamps

Timing can be everything when you’re trying to understand what went wrong with an application. You want to know the answers to questions like:

• “Was this happening before or after my database connection died?”
• “Did the crash occur before or after I ran out of disk space?”
• “Exactly when did that request come in?”

The Python logging library can help you add timestamped information into your log without you having to call ‘datetime.datetime.now()’ all the time. So, please don’t do that. Please use the timestamps provided by the logging library instead.

You Know Your Logs’ Origin

The logging library that Python provides also helps you gather context about where a particular log message was emitted. You can capture which module it came from, even down to the function and line number in your code. If you’re running something that’s multi-threaded, it can tell you from which thread the log message was emitted.

In one case, I was able to use this type of context to determine that I was actually using the wrong Python library. I had a copy in the system site-packages and I had a copy in my virtualenv. By analyzing my log data, I figured out that even though I wanted the one in the virtualenv, I was actually importing the version from site-packages.

Here’s an example with both timestamps along with file and line number information:

I didn’t have to add much code here. You’ll notice that I still call ‘warn’ and ‘critical,’ and the logging framework injected all of this context for me.

Configuration over Code

In my last example, I did a lot of stuff in code; I was telling the logging library exactly what I wanted. If you’re working on a Python application that involves more than a couple of modules, configuring all of that imperatively gets kind of tiresome. Thankfully, the Python logging library also includes a way for you to declaratively provide this configuration.

Configuration also allows you to easily change logging behavior when moving between different deployment environments. For example, you may want debugging logs to print to the console while in development, but only info- to critical-level logs to be saved to a file when your application is running in production. 

You create a Python dictionary that defines the formats, handlers, and destinations for specific types of logs. The dictionary tells the Python logging framework how to wire all the available pieces to get the desired logging behavior. The pieces include:

• Formatter: Responsible for taking a log event and putting it into a given text format.
• Handler:  Responsible for taking that stream and moving it somewhere. The Python logging library includes several handlers. SyslogHandler and HTTPHandler are of particular interest to Loggly users because they are ways that you can send logs to Loggly without employing any agents:
  • StreamHandler sends logs to a stream-like object, for example to a console via stdout.
  • FileHandler directs log events to a file.
  • SyslogHandler directs logs to the syslog daemon on your system.
  • HTTPHandler lets you send logs over HTTP.
  • NullHandler sends your logs to nowhere–useful for temporarily stopping logs.
• Filters: With filters, you can choose to prevent certain messages from going through or to modify the information in messages. For example, you can save only info-level or higher messages to disk or to prevent user IP addresses from being logged to disk.
• Loggers: What ties it all together. Loggers are the main API to the logging framework. That’s how the warn, debug, and info methods are exposed to you.

Here’s an example:

This configuration dictionary needs to be defined only once, then it gets imported as many times as needed throughout your application.

Using the Python Logging Library Yourself

The Python logging library has great documentation that helps you with the basics and provides code examples for trivial and non-trivial things. I find the Logging Cookbook particularly useful.

But remember: To take your logging visibility to the next level, you also need to have an effective means to know what’s in your logs and what the messages you’re getting mean in terms of your Python application’s health. If you want a much better way to see the forest for the trees as your log volumes grow, sign up for a Loggly free trial.