Why JSON is the best application log format … and how to switch
In recent years the JSON format has emerged as the de facto standard for message passing. It is both readable and reasonably compact, and it provides a standardized format for structuring data. Almost every programming language can parse it, so it’s simple to implement in languages which have not yet implemented built-in JSON functionality. JSON’s Unicode encoding makes it universally accessible, and its large and established user base provides an active community of helpful examples, patterns, and support.
In this blog post, we will examine the process of converting your log format into the JSON format by giving several examples in popular languages.
The biggest benefit of logging in JSON is that it’s a structured data format. This makes it possible for you to analyze your logs like big data. It’s not just readable text, but a database that can be queried. This allows summaries and analytics to take place and help you monitor your application and troubleshoot issues faster.
Audience and Environment
How to Switch Your App to Log in the JSON Format
In this section we will show you how to configure several common applications to log in JSON format.
For further reading on using Winston and other Node.js logging tools with JSON logs and Loggly’s service, as well as more example code and output, check out these articles from our Ultimate Guide to Logging:
Ruby: Configuring Your Sinatra App to Log in the JSON Format
Hitting the /test2 route on your localhost when running Sinatra will result in the following JSON message being sent to Loggly:
This example assumes you have the logglier gem installed globally on your system. Read more about configuration on our site.
Java: Configuring Your App to Log in the JSON Format with Log4j
Using Log4j, we can configure JSON output as described here: https://logging.apache.org/log4j/2.x/manual/layouts.html#JSONLayout. JSON can also be used for the configuration file instead of the default XML format, if preferred: https://logging.apache.org/log4j/2.x/manual/configuration.html#JSON. (Note that you’ll need to install the Jackson Data Processor to parse the JSON files if you use JSON for your config instead of the default XML.) The Log4j example file:
The Log4j configuration file (as XML):
For further information on logging with JSON and Loggly in Java, check out this article from our Ultimate Guide to Logging: https://www.loggly.com/ultimate-guide/java-logging-basics/#json. (And thank you to Andre Newman for contributing the Java code and config!)
PHP: Configuring Your Laravel App to Log in the JSON Format
Laravel has gained popularity in the LAMP world in recent years and is becoming the framework of choice for many teams. Here is an article on logging JSON logs from Laravel to Loggly from 2014.
Web Server Logs
The default log format for Apache and Nginx logs are more compact than logs in JSON, but if you need the parsing flexibility afforded by logging in the JSON format for your web server logs take a look at these examples:
Configuring Apache to Log in the JSON Format
If you’d like to see an example of configuring Apache to output JSON directly, check out this article from 2013 on Loggly’s community site.
Switching Nginx to Log in the JSON Format
Take a look at configuring Nginx to output the JSON format.
Making Sure Your JSON is Valid
It’s a good idea to use a tool such as https://github.com/zaach/jsonlint to check your JSON data. You can test the output of your new logging format and make sure it’s real-and-proper JSON.
What To Do With The Logs
Now that the logs are in JSON format, we can do powerful things with them. Many languages allow the creation of objects from JSON, and popular NoSQL databases store the format natively, but the most relevant application for our purposes may be aggregating our JSON data in a service that gives us powerful reporting, searching, and insights to our data. You can sign up for a free account on Loggly.com and send your JSON logs (among other formats) to the servers to observe trends about your data and receive alerts when certain thresholds are hit.
Don’t forget to look at examples outside of your chosen language if you get stuck. You can often convert bits of existing code from another language and convert them into your own custom solution.
Navigating JSON Logs with Loggly Dynamic Field Explorer
Once you send your logs to us in JSON format, Loggly will automatically parse your JSON and visualize its structure in a tree-like format. This makes it much easier to see what’s in your logs, figure out where to look for a problem, or spot anomalies.
While tabs on a single line provide the most compact logging format, JSON provides the most flexible one. It is relatively compact (compared with, say, XML). Most importantly, it allows for very flexible machine-first parsing.