UPDATE: I’ll be doing a slightly expanded version of this talk at Lucene Revolution in Boston on October 8th, incorporating some of the stuff I talk about below.

I got a few interesting questions and comments after the talk, so I thought I’d expand a bit on what was in my slides, which were perhaps a little dense.

“Log Search is highly skewed”

In the talk, I said that the most important search data is the most recent. When you have a problem, you’re far more likely to care about what happened in the last few minutes or hours or days than what happened a month ago. Thats not say that you’ll never need to search older data, just that most of the time, you won’t.

After the talk, though, it became obvious that I should also have said that our users are likely to use search in a way that is also pretty skewed when compared to “normal” search products. Basically, we expect that most people will use the system somewhat sporadically, but that when they do, its likely to be a pretty intensive session of  bug hunting. So instead of a fairly continuous search load, we get random spikes for a small subset of all the data we have in Solr. This is actually good for us, because we don’t need to keep all of the shards for all of our customers “hot” in Solr.  When a customer shows up, we can warm their data quickly, and let Solr and the filesystem cache do their thing to deal with shards that haven’t been used for a while.

The most important point here is that the overall system is going to be spending the vast majority of its resources on indexing, rather than searching. I can’t give you numbers, but if we end up spending anything more than about 5-10% of our cycles on search, I’ll be very surprised. This is not your typical consumer search product.

0MQ

I talked a bit about 0MQ, and said that we chose it primarily because its fast and lightweight, even though its possible that we could lose data if things break. I clarified this a bit in a comment on Sarah Allen’s blog because I want to make sure the message is that 0MQ is awesome, not that it loses data. Here’s the guts of what I said…

I wanted to clarify one point in your writeup, though, to make sure people don’t get the wrong idea about 0MQ. Yes, our implementation of 0MQ has a potential “leak”, where we can lose messages, but its a very uncommon case, and the impact is small. Specifically, if one of the solr nodes dies hard, we potentially lose any events that were sent to it in the last batch (0MQ batches to minimize comms overhead). In steady state, 0MQ is rock solid, 100% reliable, and faaaaaast.

Pieter (at iMatix) and I are currently discussing ways to solve the hard death problem, and I don’t anticipate it being a problem very long. As I said in the talk, 0MQ is unbelievably cool – if you haven’t got a project that needs it, make one up!

We sponsored some work to get the SWAP functionality in version 2 of 0MQ, and I’ve been blown away by the guys at iMatix – they really want 0MQ to work, and work well. My  throw-away comment prompted an email from Pieter asking for more details, and, as I said to Sarah above, we’re already looking at how to fix it.

Oh, and in case you’re wondering how fast a one-armed paper-hanger is, take a look at what The Word Detective says about it (scroll down till you see the “You missed a spot” section). Maybe I should have used “flat out like a lizard drinking” instead?

Sharding

The way we create shards by indexing, then merging, then merging again and again and again raised a few questions that are worth repeating…

To recap, we build small (5 minute) shards on our hot indexers. When we stop adding events to them, they get merged with older shards until we hit another size limit (30 minutes). They then get merged with even older shards, until we hit the next time limit (4 hours). And so on up the chain until they cap out at a week long. Along the way, we push indexes from box to box, to balance the load on the system as a whole.

The first question is fairly obvious: Why?

At first glance, it seems like we’re just creating work for ourselves. Surely we could just build the shards and use them as is, right? The problem is that we would have a lot of 5 minute shards floating around the system, and we already know that Solr starts getting cranky when you run a lot of cores in a single instance. So, why don’t we just build bigger shards? The issue there is that with the version of Solr we’re using, we have to reopen the index to make new data available, and we currently do that every 10 seconds (hence the “NRT + SolrCloud = Our Nirvana” in my slides). Since we have to do this, we’d end up with too many segments in the hot index, or (if we’re not careful with our merge factor) a lot of automatic merging that means that the hot index becomes unavailable for updates for too long for my liking. So, we got pushed into this approach by something that I’m hoping will soon be a thing of the past. I’m really looking forward to Michael Busch’s talk at Lucene Revolution which promises to remove the “N” from NRT. I’m not sure what is better than nirvana, but I’m hoping to find out soon

We may have been forced into doing things this way, but there is a lot of value in the model we have. In some ways, we’re taking over a part of Lucene (merging) that has been absolutely  invaluable, but can sometimes be a little difficult to control. We now have complete control over when and where indexes get merged. I probably should point out that we deliberately don’t do any merging on the 5 minute shards, and that we’re careful with the merge parameters on the larger shards to make the merges that do happen as efficient as possible.  The model also gives us a very simple index naming scheme based on time, which means we always know exactly where to find data for a time-constrained query. More on this in a bit…

The next question (from the meetup) was what is the overhead of all this merging?

Rather than give numbers, its worth thinking about whether we’re actually doing anything more than Lucene already does when you start building big indexes. I think the answer to that is that we’re actually just exposing and taking over the automatic behaviour, rather than doing something “extra”. So I think the real overhead is close to zero. Compared to building a bunch of shards in parallel using Hadoop, we’re certainly doing more work, but most of the Hadoop based systems I’ve looked at are geared more towards building indexes from a large existing corpus, rather than dealing with a real time stream.

My final comment on this is that since its all completely configurable, we’re not locked into any of the times I’ve mentioned above. Maybe when we move to NRT, or RT, we can bump the hot shard size up to hours or days, assuming that we’re still in control of merging. We shall see…

Constructive Laziness

Circling back to the first section, where I talked about how skewed we expect our search to be, the time-based shards gives us a very clean way to limit the impact of our search requests. Since we can constrain a search to a specific time period, its easy for us to identify which indexes we need to hit to satisfy the search. Our ideal search is for something in the last few minutes, which can be entirely served out of one or two of the five minute shards. We may have gigabytes or (hopefully) terabytes of index data for the same customer sitting around on our system, but if we can satisfy their request by hitting two small, heavily cached cores, then we’re in great shape. I wonder if life will be so kind to us?

Random aside: Synchronicity

Every now and then, things just come together in strange ways. A couple of weeks ago, Kord and I talked with Diego and Santiago from Flaptor, who are working on IndexTank. Diego and I were at LookSmart together years and years and years ago, but thats not the synchronicity. As we were talking, Diego said they were working on a “Nebulizer” which does automatic distribution of their index in the cloud. The day before the meeting, I’d pulled all of the code that deals with this in our system into a class named “TheDecider” (I’m still wrestling with a way to make misunderestimate() a useful method in this class). That evening I went to a NoSQL meetup, and met someone who is also working on the equivalent for their system. Maybe there is something in the air?

Well, this seemed wrong. Why did we just design a really nice, RESTful API and then developers who want to use it have to build a server-side wrapper first. This didn’t make sense to me. So I kept digging. Fortunately, I found the solution. It’s called JSONP (JASON with Padding). Here is how it works and how you can leverage it in your own applications.

Let’s assume I am building an application at labs.loggly.com that will access the API located at loggly.loggly.com. With jQuery, my AJAX call looks as follows:

$.ajax({url: "http://loggly.loggly.com/api/search/?q=ntp", username="guest", password="loggly", ...})

Now, if you do this, you will get the cross-domain error. However, if you just slightly change your call to include an extra parameter, it will succeed:

$.ajax({url: "http://loggly.loggly.com/api/search/?q=ntp",
    username='guest', password='loggly',
    dataType:'jsonp',
    success: function(data) {
        flare = data['data'];
    },
    error: function(XMLHttpRequest, textStatus, errorThrown) {
        alert(textStatus+" - "+errorThrown);
    }
})

Note the newly added dataType parameter. That’s it? Yes, that’s it. It will work like a charm. No more cross-domain security issues. What basically happens are two things. First, the AJAX request that is executed has one more extra query parameter: &callback=?, where the question mark is some string that jQuery randomly generates. The second thing that happens is on the Loggly side. If the callback parameter is present, Loggly does not return the plain JSON element that you would expect, but it wraps it in a function call. Something like:

jsonp12312312({data:{"May-20-2010 12:13:45": 2"}, numFound: 1})

The next thing that happens is that when your browser gets the answer back like this, it will try to execute the function called jsonp12312312. jQuery internally handled that for you by creating a function hook for that function that points to the success function provided to the AJAX call.

That’s really it. We are looking forward seeing your applications that are using the Loggly APIs!

By the way, Loggly is using Django Piston for handling the APIs. The library automatically handles JSONP responses when a parameter called “callback” is present!

Here is how we log from within our application:

from loggly.logging import *

error({'object':'input','action':'create'})

That’s it. The above code creates the following log entry:

Mar 18 15:34:03 app loggly: severity=ERROR,user=logdog_zrlram,request_id=
08BaswoAAQgAADVDG3IAAAAD,object=input,action=create,status=failure

The logging call expects a dict of key-value pairs. This is to enforce key-value based log entries that make it easy for consumers to understand what a specific value means. Without the inclusion of a key, a value is more or less useless. In the example above, note that I only provided two keys: object, and action. However, the log entry contains a number of other data items. Those items are automatically added to the log entries by our logging library without burdening the developer to explicitly include them.

It is probably time to show you Loggly’s logging library:

import logging
import inspect

DEFAULT_LOGGER = 'loggly_web'
logs = None

def logHelper(rest=None, request=None):

    global logs
    output = list()

    # get the logger
    if not logs:
        logs = logging.getLogger(DEFAULT_LOGGER)

    # Loop through all the stack frames until you find the request
    stack = inspect.stack()
    for frame in stack:
        if frame[0].f_locals.has_key('request'):
            request = frame[0].f_locals['request']
            if request is None:
                continue
            # there is a request object
            if hasattr(request,'user') and hasattr(request.user,'username') and len(request.user.username)>0:
                output.append("user="+str(request.user.username).strip())
            if hasattr(request,'META') and request.META.has_key('UNIQUE_ID'):
                output.append("request_id="+str(request.META['UNIQUE_ID']).strip())
            # we found the request object. Get out of here
            break

    # getting input dictionary and appending
    if rest:
        for key in rest:
            output.append("%s=%s" % (str(key.strip()), str(rest[key]).strip()))

    ret = ",".join(map(str, output))
    return ret

def info(rest=None, user=None):

    msg = logHelper(rest, request)
    logs.info(msg)

def error(rest=None, user=None):

    msg = logHelper(rest, request)
    logs.error(msg)

Note that this is only an extract. Download the entire library if you want to use it in your own code. Here are some important things the code does:

• line 17 to 29: This part of the code inspects the call stack to check whether there is an HTTP request object somewhere. The request object contains the username for the session and that is what we automatically extract . This frees the user from manually adding that information to the logging call. Automation is good!
• line 26 and 27: We are using UNIQUE_IDs in Apache. In order to track a request from the Apache logs down into our application, we include that same ID into our Django logs. This is a huge win for associating Apache logs with our application logs.
• line 32 to 24: All the dict entries are added as ‘key=value’ pairs to the log entry. So you can log any key you want.
• line 39 to 47: These are the calls that you use in your code. Note that you can add a user field, which overwrites the username from the request. In some cases that is necessary and useful.

Let us know if you are using our library. I would love to hear back from you. I will post another blog entry later, where I will be talking about how to patch Django itself to do some more logging. We will be looking at how the authentication methods can be extended.

The links:
Django 1.2 Logging Patch
Loggly Logging Library

tar -xzf Django-1.2.tar.gz
cd Django-1.2
patch -p0 < /tmp/django_logging_1.2.patch
python ./setup.py install

This will include the patch into your Django distro and install it. As a next step, you configure your application for logging by adding a snippet similar to the following in your settings.py file:

LOGGING = {
    'loggly_feature': {
        'handler': 'logging.handlers.SysLogHandler',
        'address': ('localhost', 514),
        'facility': 'local3',
        'level': 'INFO',
        'format': 'loggly: %(message)s',        # does not include the severity. it's features, baby!
    },
}

From there you use the following code snippet to log out of your application:

import logging
logs = logging.getLogger('loggly_feature')
logs.error("message")

More information about how to use the logging libraries you find in the Python documentation. It also seems like Django 1.3 will have logging built in. From what I can tell, it looks similar to Simon's approach, but it's not quite the same. I'll keep you posted here!

In my next blog post I will show you how we implemented a logging library for Loggly so that we can very easily log from anywhere within our application.

Django 1.2 Logging Patch

The other day I was reading about the Apache and Atlassian hack. Max wrote a really nice summary of how that attack could have been prevented. One of the points he raised was that they should have used HTTPONLY cookies.

I then realized that we might have the same problem with Loggly. After some traffic dumping of our Web sessions, I realized that Django didn’t support httponly cookies. A quick google search revealed that someone wrote a djangosnippet to add httponly cookies. I had to slightly rewrite it, so here is the code I am using:

class cookie_httponly:
    def process_response(self, request, response):
        scn = settings.SESSION_COOKIE_NAME or 'sessionid'
        if response.cookies.has_key(scn):
            response.cookies[scn]['httponly'] = True
        return response

Don’t forget to add the middleware right before the SessionMiddleware. If you are using Python 2.6 or higher, you are done. Unfortunately, we are running Python 2.5, which does not support the httponly flag on cookies. A quick patch solved that problem as well:

--- /usr/lib/python2.5/Cookie.py   (revision 66233)
+++ /usr/lib/python2.5/Cookie.py   (working copy)
@@ -408,6 +408,9 @@
     # For historical reasons, these attributes are also reserved:
     #   expires
     #
+    # This is an extension from Microsoft:
+    #   httponly
+    #
     # This dictionary provides a mapping from the lowercase
     # variant on the left to the appropriate traditional
     # formatting on the right.
@@ -417,6 +420,7 @@
                    "domain"      : "Domain",
                    "max-age" : "Max-Age",
                    "secure"      : "secure",
+                   "httponly"  : "httponly",
                    "version" : "Version",
                    }

@@ -499,6 +503,8 @@
                 RA("%s=%d" % (self._reserved[K], V))
             elif K == "secure":
                 RA(str(self._reserved[K]))
+            elif K == "httponly":
+                RA(str(self._reserved[K]))
             else:
                 RA("%s=%s" % (self._reserved[K], V))

Loggly is now more secure against XSS attacks!

A short while into writing code for the Loggly interface we decided that we needed some eye candy. Given my background in visualization, I was keen on providing our users with an experience that helps them understand their data in an intuitive way.

Over the last few years I’ve been looking into a ton of visualization libraries for the Web. In the past, if you had asked me what library to use for generating charts on your Web site, I would have said, “Use Flash”. While there are a number of interesting Flash libraries out there, the landscape has shifted significantly in the last year. Everyone is moving to JavaScript. After some research, I opted to use a JavaScript charting library called HighCharts. I tried a bunch of other canvas-based libraries, but let me tell you without hesitation, HighCharts rocks.

I am going to show you how we are using HighCharts and how I implemented zooming to dynamically reload more event data on the fly. With any charting library, if you keep zooming in on a chart, it will not progressively load more detailed data. At detailed zoom levels you end up with a small range of data in your graph. Basically if you view a day’s data first, and then zoom into a specific minute, you would only see one data point.

To start, here’s the JavaScript I use to display a chart:

var parse_date = function(data) {
    var result = [];
    $.each(data, function(key, value) {
        var re = new RegExp(/(\d+)-(\d+)-(\d+)T(\d+):(\d+):(\d+)(?:\.(\d+))?/);
        var date = re.exec(key);
        if (date[7] == undefined) {date[7]=0;}
        var real_date = Date.UTC(date[1], parseInt(date[2])-1,date[3],date[4],date[5],date[6],date[7]);
        result.push([real_date, value]);
    });
    return result;
}

chart = new Highcharts.Chart({
    credits: { enabled: false },
    chart: {
        renderTo: 'activity',
        defaultSeriesType: 'area',
        margin: [10, 20, 40, 55],
        zoomType: "x",
            events: {
                selection: function(event) {
                    // change the time frame to be searched
                    var start = Highcharts.dateFormat('%Y-%m-%dT%H:%M:%SZ', event.xAxis[0].min);
                    var end = Highcharts.dateFormat('%Y-%m-%dT%H:%M:%SZ', event.xAxis[0].max);
                    $.ajax({ type: "GET", url: "http://subdomain.loggly.com/api/search/?" \
                        + "q=inputname:logglyapp&starttime="+start+"&endtime="+end \
                        + "&facets=True&buckets=24",
                        success: function(data) {
                             chart.xAxis[0].setExtremes();
                             chart.series[0].setData(parse_date(data));
                             // fix the reset zoom button
                             $('.highcharts-toolbar').click(resetZoom);
                        },
                        error: function(req, text, error) {
                            $("#err").html("Reload error!");
                        }
                    });
                }
        }
    },
    xAxis: { title: { text: 'Time' }, type: 'datetime' },
    yAxis: { title: { text: '# Events' }, min:0,
        plotLines: [{ value: 0, width: 1, color: '#808080' }]
    },
    tooltip: { formatter: function() {
            return Highcharts.dateFormat('%B %e %Y %H:%M:%S', this.x) + '
'+
            ''+this.y+' Events' }},
    plotOptions: {
        area: {
            dataParser: parse_date,
        }
    },
    series: [{ id: 1, name: 'search',
        dataURL: 'http://subdomain.loggly.com/api/search/'
            + '?q=inputname:logglyapp&facets=True'}],
    title: { text: 'traffic last 24 hours' }
});

var reset_zoom = function() {
    // requery for the original data:
    $.ajax({ type: "GET", url: "http://subdomain.loggly.com/api/search/"
        + "?q=inputname:logglyapp&facets=True",
        success: function(data) {
           chart.toolbar.remove('zoom');
           chart.xAxis[0].setExtremes();
           chart.get(1).setData(parse_date(data));
        },
        error: function(req, text, error) {
            $("#err").html("Loading error!");
        }
    });
}
});

Let’s have a quick look at the code. There are two things I want to communicate here: 1. The code I used to display a HightChart graph and 2. The way I am using Loggly’s APIs to query the data.

I mentioned the special zooming that I implemented. Take a look at lines 20 to 39. This is the function that handles zooming, and it is where I am reloading the more detailed data. I set the new start and end dates (lines 23 and 24) and then I am querying the Loggly API with the new timeframe (lines 25 to 27). Upon success – this is important – I am using the chart.series[0].setData() method to set the new data for the chart. The next line overwrites the default button or a link that lets the user zoom out again (lines 32). Note: because you are implementing your own zoom, the default “reset zoom” button from HighCharts will not work anymore and you have to implement your overwrite it with your own function to reset the chart.

The function dealing with the reset functionality is on lines 59 to 72. It does nothing else than query the Loggly API for the original data (I am passing no time parameters) and setting the data just like the previous call. The other thing you have to do is in lines 64 where you need to remove the HighCharts default “reset zoom” link and reset the extremes (line 65).

Moving on, we’ll briefly discuss the way I’m using the Loggly API. If you’d like to use it, you need an account with us. We are currently in private beta, therefore you will need us to give you access to the beta program in order to do so. Email if you want an account to play around with! Back to the code. Make sure you replace the with your actual subdomain. Now that this is out of the way, you can query the API by simply making a GET request to: /api/search. You pass the q parameter with your query. In my example I am getting all the data from my input with the name logglyapp. To get timeline data, you’ll need to pass the parameter facets=True into the call. This will give you counts for time buckets.

To make everything work together, you need one more piece: the date_parse function. You need this part because the Loggly API returns the data with real human readable timestamps and HighCharts wants UTC encoded timestamps. The function on lines 1 to 11 takes care of converting the time for you. Just copy it.

I hope this was useful. Let us know if you are having trouble with any of this. We are looking forward hearing about your graphing endeavors.

If you look at my del.icio.us feed, you’ll find a bunch more visualization and charting links.

If you are running your Web servers behind a load balancer, you have probably noticed that your logs contain the load balancer’s IP address as the client IP, which is kind of annoying. There is an Apache module called RPAF, which fixes exactly this issue. Once you have it downloaded and installed, you configure it as follows:

RPAFenable On
RPAFsethostname On
RPAFproxy_ips 127.0.0.1 10.0.0.1
RPAFheader X-Forwarded-For

The module works such that it takes the IP address that is being transmitted in the X-FORWARD-FOR header and sticks it into the request as the ClientIP.

The not so nice part is that the RPAFproxy_ips statement lets you define the IP addresses of your load balancer. If you have only a set amount of them, all is fine. But here comes the catch. If you are deployed on Amazon AWS and you are using their load balancer (LB), then this solution is going to frustrate you quite a bit. You will notice that the LB’s IP address changes constantly and you keep adding IP addresses to the configuration statement. After about 10 IP addresses, I got sick of that and I started looking at the source code of RPAF to solve this problem once and for all. Here is what I did:

On line 139 of mod_rpaf-2.0.c, I added a return 1; statement. This will tell the is_in_array() function to always assume that the request is coming from a load balancer, without checking the configured list of IP addresses. The rest of the RPAF code is robust enough to only replace the client ip when an X-FORWARD-FOR header is actually set. After the change, do a make install-2.0 and you are in business.

Happy logging!

What I want to focus on today is our integration with RightScale. At Loggly, we use RightScale to manage our AWS instances. Loggly runs three types of servers. (Well, I am simplifying). We have a proxy tier which receives your log messages. The proxy tier, which is basically a bank of machines, forwards the messages to the indexing back end that runs Solr. The third group of machines are the Web or application servers. When a new proxy box comes online, the RightScale management interface knows about the box. I had to know about thse proxies on the application tier (i.e., within Django) as well. How do you do that?

The first solution would be to have the proxies register with Django, as soon as they get online. What happens though when they go down or are taken offline? Seems complicated to keep track of that. Another solution would be to periodically poll the proxies from Django. Not very nice either.

My solution is much more elegant. RightScale has two features that helped me out. The first one is machine tags. Each proxy server is labeled as such. (See Machine Tagging). Secondly, I am using the RightScale API to figure out how many proxies I have and what their IPs are. (As a side note, the RightScale APIs are in Beta right now. There might be changes or improvements coming down the pipe.)

I struggled for quite a bit with using the RightScale APIs out of Python. Here are some things that I learned the hard way and you might find helpful:

Using the API to query all your machines in a specific deployment:

curl -H 'X-API-VERSION: 1.0' -u [user@domain.com]:[password] \

https://my.rightscale.com/api/acct/[account]/deployments/[deployment_number]

Note how you have to add the extra header to request version 1.0 of the API.

Here is how you get all the machines that have a specific tag. Note the structure of my tag! I set role:proxy=true. You need to use this hierarchical model!

curl -H 'X-API-VERSION: 1.0' -u [user@domain.com]:[password] -d'resource_type=server' \
-d 'tags[]=role:proxy' https://my.rightscale.com/api/acct/[account]/tags/search.js

Want JSON output instead of XML, add “&format=js” at the end of your request!

Now, from the response, you would think you could just use that HREF to query an individual server. Wrong. That doesn’t work. You have to add “/settings” in order to make that work:

curl -H 'X-API-VERSION: 1.0' -u [user@domain.com]:[password] \

https://my.rightscale.com/api/acct/20184/instances/[instance_id]/status

Here is how you set a tag on a server: (Note: If you change the tag in the user interface for a running server, it will not take effect. Only if you start a new server of that type, will the tag be there. Unlike the API call, where you can set a tag on a running machine).

curl -H 'X-API-VERSION: 1.0' -u [user@domain.com]:[password] \
-d 'resource_href=https://my.rightscale.com/api/acct/[account]/servers/[server_id]' \
-d tags[]=role:proxy=true https://my.rightscale.com/api/acct/[account]/tags/set

The part I struggled with most was how to call the API from within Python. Turns out httplib2 expects the Web server to respond slightly different than the RightScale server is. If you are using the following code, you will not be able to connect:

h = httplib2.Http()
h.add_credentials(user,password)
response, content = h.request(url, headers=headers)

httplib2 will connect to the Web server without sending the credentials. Only if the server challenges the client to use auth, it will then send the authentication headers. And this is precisely what RightScale is not doing. Therefore, you have to do the following in order to include the authentication headers in the first request already:

h = httplib2.Http()
import base64
base64string = base64.encodestring('%s:%s' % (user, password))[:-1]
headers['Authorization'] = "Basic %s" % base64string
response, content = h.request(url, headers=headers)

Credentials are an interesting topic. I ended up creating a separate user in the RightScale interface that I am using for the APIs. Don’t be fooled though. These credentials still let that user log into the Web interface. I hope that RightScale will add a capability such that I can have a user that can only use the API.

I hope this helps you getting off the ground a bit quicker when using RightScale. Let me know how it goes. You can also find me on Twitter: @zrlram

One of the things you’ll notice when you use Unfuddle is the presence of a subdomain in the URLs you use on the site.  Our subdomain is ‘loggly’ on Unfuddle, and we  log into our project area by going to http://loggly.unfuddle.com/ (no, you can’t check our code out).  This type of customer segmentation allows for multiple unique usernames per customer, but doesn’t require a unique username site-wide.  For non-SEO sections of the site, this is a perfect solution.

We are taking a similar approach with Loggly, where a user will sign up for an account and define a unique customer identifier (we’re kicking around calling this a “mill”), which will then be mapped to a subdomain on the system.  So, for example, if Foobar, Inc. were to sign up for a Loggly account, they would access the site via http://foobar.loggly.com/, and then could create any number of user/pass combinations they wanted to access their company’s log resources.

The only problem with this approach is that we use Django, and their built in auth system (which is fantastic, BTW) doesn’t really have facilities for this type of functionality.  While we could certainly hack the Django auth system by writing our own multi-tenant auth module, it would take away from more pressing issues – like launching the beta!

Enter the Middleware Solution

One way to solve this is by munging the subdomain and username together, which provides a unique system-wide username. If, for example, you were to log in as steve under foobar.loggly.com, then we’d stick them together to be something like “foobar_steve”. Obviously we can’t have everyone remembering this long monstrosity for their username, so we’ll need to munge the subdomain off the URL and the username the user types in to get the correct combination to send off to the auth system.

Thankfully Django provides a super-easy way to add middleware to a project. By injecting a small piece of code into the request from the user’s browser, we are able to do our on-the-fly transformation before the auth system takes over. Nobody is the wiser because we can modify the display name code in the profile model to show the “normal” username to the user. Here’s what the result looks like:

settings.py:
...
MIDDLEWARE_CLASSES = (
'loggly.profile.MungeMiddle.MungeForMillMiddleware',
...
)
...

MungeMiddle.py:
class MungeForMillMiddleware:
    def process_request(self, request):
        if request.POST.has_key('username'):
            data = request.POST.copy()
            user = "%s_%s" % (request.META['HTTP_HOST'].split('.')[0], data['username'])
            data['username'] = user
            request.POST =  data

When a request comes in, we pull out the POST data and make a copy of it with .copy(). We then munge up the username with the subdomain out of HTTP_HOST, and then set the POST data to forward on to the rest of the stack. We don’t do this for all requests, just ones with the username set, so it’s lightweight enough for production use. We end up sticking the shorteded version of the username into the profile table, and use it for display.

So there you have it. A 5 minute fix for a 5 hour problem. I’m sure there are more elegant solutions to doing subdomain segmentation with Django’s out-of-the-box auth system, but frankly we don’t have time to stop and code them up. We’re bent on getting our beta out as soon as possible, and if it requires hacks like these to do it, then so be it! Release early, release often.

Last year I did a coolcam mashup based around Poly9’s Flash Globe and events from my web server’s log files. I never did get around to publishing the code, but the URL was accessed 100s of times by the sales guys I worked with. If it ever was down, or broken, I’d get an email from then in minutes. As it turns out, a lot of them used the globe to start conversations with their customers.

Loggly Globe
Flash forward to present day. I’ve completely rewritten the code and put it up on Loggly’s site to share with everyone. The way the globe works is pretty simple. Using the web.py framework, it starts a web server which does two things. First, it serves up the HTML to your browser, which includes the Poly9 globe object and the jQuery library. Second, it serves up a JSON object to the page which is parsed and sent to the globe object. The code that serves up the JSON object does a few magical things for you:

• tails yor web access log file for visits
• parses out the ip address, timestamp, etc. from the log event
• takes the ip address and does a geoip lookup on it
• removes duplicate visits from a single ip address
• wraps the whole thing up in a JSON object

While Loggly Globe is hard coded to parse our logs, it should be fairly easy to use it yourself. To get started, download the tarball for Globe 1.0, and then extract it somewhere on your server:

kord@loggly> tar xvfz globe_1.0.tar.gz

You may need a couple of Python libraries installed. Assuming you have easy_install installed you can run:

kord@loggly> easy_install web.py
…
kord@loggly> easy_install httplib2

You’ll want to edit the globe.py file and modify the location of the Apache log file to point to your local log file. You’ll also want to edit the regular expression extractions to match your log file format. Here’s a line out of our logs for reference, and the corresponding extractions, most of which were pulled from Random Encounter. Make any changes you need to match the regex up with your logs.

75.101.142.96 - - [11/Sep/2009:09:19:17 -0700] "GET / HTTP/1.1" 200 6196 "-" "collectd/4.4.2" 195546

parts = [
   r'(?P\S+)', # host %h
   r'\S+', # indent %l (unused)
   r'(?P\S+)', # user %u
   r'\[(?P.+)\]‘, # time %t
   r’”(?P.+)”‘, # request “%r”
   r’(?P[0-9]+)’, # status %>s
   r’(?P\S+)’, # size %b (careful, can be ‘-’)
   r’”(?P.*)”‘, # referer “%{Referer}i”
   r’”(?P.*)”‘, # user agent “%{User-agent}i”
   r’(?P[0-9]+)’, # stuff at end
]

Now you'll want to start the server. You can specify a port number to listen on if you want:

kord@loggly> cd globe
kord@loggly:/globe> python globe.py 8001

http://0.0.0.0:8001/

Try hitting http://yourserver:8001/json and see if you get a response back. Here's an example of what you should see: http://www.loggly.com:8001/json. Here's the demo again, if you just want to skip to the good stuff. Additional work could be done to integrate the code into an Lightty or Apache install to make it more permanent. You can read more about doing that on Web.py's cookbook page.

Once we get the beta launched, you'll be able to make mashups like these with your own log files. We're looking forward to doing more coolcams like this with Loggly!