Coolcam for Your Log Files


I’ve been talking about coolcams for years as a way to help quickly show off a product’s features. Coolcams aren’t meant to be useful – they exist simply to entertain and engage your audience. It’s like an elevator pitch for your product demo.

Last year I did a coolcam mashup based around Poly9’s Flash Globe and events from my web server’s log files. I never did get around to publishing the code, but the URL was accessed 100s of times by the sales guys I worked with. If it ever was down, or broken, I’d get an email from then in minutes. As it turns out, a lot of them used the globe to start conversations with their customers.

Loggly Globe

Flash forward to present day. I’ve completely rewritten the code and put it up on Loggly’s site to share with everyone. The way the globe works is pretty simple. Using the framework, it starts a web server which does two things. First, it serves up the HTML to your browser, which includes the Poly9 globe object and the jQuery library. Second, it serves up a JSON object to the page which is parsed and sent to the globe object. The code that serves up the JSON
object does a few magical things for you:

  • tails yor web access log file for visits
  • parses out the ip address, timestamp, etc. from the log event
  • takes the ip address and does a geoip lookup on it
  • removes duplicate visits from a single ip address
  • wraps the whole thing up in a JSON object

While Loggly Globe is hard coded to parse our logs, it should be fairly easy to use it yourself. To get started, download the tarball for Globe 1.0, and then extract it somewhere on your server:

kord@loggly>tar xvfz globe_1.0.tar.gz

You may need a couple of Python libraries installed. Assuming you have easy_install installed you can run:

kord@loggly> easy_install

kord@loggly> easy_install httplib2

You’ll want to edit the file and modify the location of the Apache log file to point to your local log file. You’ll also want to edit the regular expression extractions to match your log file format. Here’s a line out of our logs for reference, and the corresponding extractions, most of which were pulled from Random Encounter. Make any changes you need to match the regex up with your logs. - - [11/Sep/2009:09:19:17 -0700] "GET / HTTP/1.1" 200 6196 "-" "collectd/4.4.2" 195546

parts = [
   r’(?P\S+)‘, # host %h
   r’\S+’, # indent %l (unused)
   r’(?P\S+)‘, # user %u
   r’\[(?P.)\]‘, # time %t
   r’"(?P.)"’, # request “%r”
   r’(?P[0-9])‘, # status %>s
   r’(?P\S)‘, # size %b (careful, can be ’-’)
   r’"(?P.)"’, # referer “%{Referer}i”
   r’"(?P.)"’, # user agent “%{User-agent}i”
   r’(?P[0-9]+)’, # stuff at end

Now you’ll want to start the server. You can specify a port number to listen on if you want:

kord@loggly> cd globe

kord@loggly:/globe> python 8001

Try hitting http://yourserver:8001/json and see if you get a response back. Here’s an example of what you should see: Here’s the demo again, if you just want to skip to the good stuff. Additional work could be done to integrate the code into an Lightty or Apache install to make it more permanent. You can read more about doing that on’s cookbook page.

Once we get the beta launched, you’ll be able to make mashups like these with your own log files. We’re looking forward to doing more coolcams like this with Loggly!

Share Your Thoughts