Visualizing Your Data with Python and Loggly
Logging is critical if you want to diagnose problems or understand what’s happening with your Python application. In this post, I hope to convince you that using Python’s logging library is preferable to just putting ‘print’ statements everywhere in your code.
The Python logging library lets you:
Here’s a bare-bones example of logging in Python:
You import the logging library, you define here where you want log events to go, and then you emit a log event.
If you are looking for Python Logging 101, check out our Ultimate Guide to Logging.
With the Python library at your service, you can include several more types of metadata in your logs. By doing so, you’ll gain valuable context around your log messages to help you diagnose a problem or figure out what’s going on in your system.
Segmenting log events by severity level is a good way to sift through which log messages may be most relevant at a given time. A log event’s severity level also gives you an indication of how worried you should be when you see a particular message.
When you’re still developing your code, debug information is very useful. You want to have the most detailed diagnostic log messages available.
When you’re running your application, log info messages. These are just things that say, “I opened a socket” or “I got a request” or “The database record is inserted correctly,” for example.
Warnings are things that aren’t quite problems yet but might be signs of a future problem. For example, “The disk is 80% full” or “Latency is starting to creep up.” Warning messages tell you what to keep an eye on so that you can take care of business before a problem erupts.
Errors are things that demand immediate attention. If you see a message like, “I tried to reconnect, but the database isn’t there,” you should be worried. Now.
And finally, criticals are indications of big problems. Your program is not likely to be able to continue, and users won’t be able to use the application.
Here’s a quick example of how you can set your logging level in your logger to log only things that are error and above. So, the warning message “Citizens of Earth, be warned” is not displayed. However, the critical message “Citizens of Earth, your planet will be removed NOW” does appear in the console when you run your program.
Timing can be everything when you’re trying to understand what went wrong with an application. You want to know the answers to questions like:
The Python logging library can help you add timestamped information into your log without you having to call ‘datetime.datetime.now()’ all the time. So, please don’t do that. Please use the timestamps provided by the logging library instead.
The logging library that Python provides also helps you gather context about where a particular log message was emitted. You can capture which module it came from, even down to the function and line number in your code. If you’re running something that’s multi-threaded, it can tell you from which thread the log message was emitted.
In one case, I was able to use this type of context to determine that I was actually using the wrong Python library. I had a copy in the system site-packages and I had a copy in my virtualenv. By analyzing my log data, I figured out that even though I wanted the one in the virtualenv, I was actually importing the version from site-packages.
Here’s an example with both timestamps along with file and line number information:
I didn’t have to add much code here. You’ll notice that I still call ‘warn’ and ‘critical,’ and the logging framework injected all of this context for me.
In my last example, I did a lot of stuff in code; I was telling the logging library exactly what I wanted. If you’re working on a Python application that involves more than a couple of modules, configuring all of that imperatively gets kind of tiresome. Thankfully, the Python logging library also includes a way for you to declaratively provide this configuration.
Configuration also allows you to easily change logging behavior when moving between different deployment environments. For example, you may want debugging logs to print to the console while in development, but only info- to critical-level logs to be saved to a file when your application is running in production.
You create a Python dictionary that defines the formats, handlers, and destinations for specific types of logs. The dictionary tells the Python logging framework how to wire all the available pieces to get the desired logging behavior. The pieces include:
Here’s an example:
This configuration dictionary needs to be defined only once, then it gets imported as many times as needed throughout your application.
The Python logging library has great documentation that helps you with the basics and provides code examples for trivial and non-trivial things. I find the Logging Cookbook particularly useful.
But remember: To take your logging visibility to the next level, you also need to have an effective means to know what’s in your logs and what the messages you’re getting mean in terms of your Python application’s health. If you want a much better way to see the forest for the trees as your log volumes grow, sign up for a Loggly free trial.
Ivan Tam
James DeHart
3 years agoyour “Logging Cookbook” url is broken. great post.
Linda Somme
3 years agoThanks James – we got the link fixed.