If you're like anyone else in the world, you probably don't like wasting a lot of time scouring through log files. Even though Loggly makes log files fun, we want to help you get more out of your logs without even looking at your logs. This is where structured data comes in. Many of our users have made the transition to JSON and aren't going back!
Here's an example. Say this is a portion of your log file:
I want to figure out how many 29 year olds are from San Francisco. This may look familiar to people who are used to dealing with unstructured logs:
With the above approach, you'll end up with log entries that include any date that has “29” in it. So then you end up with an even more complicated command.
With JSON data, that complex command that no one but you understands, becomes:
You'll need to convert your plain text logs into JSON. This is usually straight forward. Within your Apache configuration file (httpd.conf), set up a custom logging format. Here are a couple of examples:
Common Log Format:
NCSA extended/combined log format:
Once you have your JSON data, you'll need to create a new input that is JSON-enabled. We'll accept JSON data over any protocol: TCP w/ Strip, UDP w/ Strip, Secure Syslog (TLS), or HTTP(S). The “With Strip” option means that we'll strip off the syslog header before we index your data. This is so that we can easily parse the JSON.