Uniq-uely Different

 

[obsolete]

We’ve extolled the merits of sending in logs that are formatted in JSON for a while now, but I’ve been holding off on this post because I wanted to have something really exciting to tell you about.

To bring the non-JSON users up to speed: you can format your log output so that it will send something like this:

{ "Name": "Hoover Beaver", "Age": 30, "Occupation": "Tree Feller", "Address": "30 Hoover Dam Road" }

Instead of the typical unstructured log events:

"Hoover Beaver" 30 "Tree Feller" "30 Hoover Dam Road"

This means that you can search for “json.age:30″ and know that you’re only searching the Age field rather than the entire event. Don’t bother learning how to awk/sed, just tell us what field you want.

Now that everyone’s on the same page, I can tell you about one of my favorite analytics features. We’ve made some enhancements to our command “uniq”. “Uniq” works by collapsing duplicates and returning a word count. For example, if I want to find out the distribution of twenty-somethings across all Tree Fellers, I can run the following command which will give me a count for every unique age between 20 and 29 across all Tree Fellers:

> uniq json.age json.occupation:"tree feller" AND json.age:[20 TO 29] count    json.age __________________________ 455 23 358 21 276 24 121 28 

This also works well with strings. If you’re interested in seeing the distribution of error messages since your last big release, try something like this:

 > uniq json.error json.version:2.1 count    json.error __________________________ 146 ERROR 55: Trees are falling in the forest 102 WARNING 21: Too many beavers in the stream 89 ERROR 29: Can't find the right river 73 ERROR 98: The dam has broken 

In the past, we used tokenized strings to report back uniq values, which made it quite difficult to filter out the useful information. Today, we’re storing fields < 100 characters as both tokenized & untokenized strings. This analytics feature is available to only JSON users right now. If you need advice on how to change your log format & make the switch, email our support team.

Try it out and let us know how it goes!

edit: If you want uniq values through our API, try using Facets. Try something like this:

https://SUBDOMAIN.loggly.com/api/facets/json.FIELD?q=inputname:INPUT%20json.FIELD:VALUE

Currently uniq search terms are limited to one within the UI. If you’d like to use the uniq command for more than one term please use the API:

https://SUBDOMAIN.loggly.com/api/facets/json.FIELD/?q=json.FIELD:TERM1%20json.FIELD:TERM2%20json.FIELD:TERM3&from=NOW-7DAYS&until=NOW


Share Your Thoughts

Top