Well, this seemed wrong. Why did we just design a really nice, RESTful API and then developers who want to use it have to build a server-side wrapper first. This didn’t make sense to me. So I kept digging. Fortunately, I found the solution. It’s called JSONP (JASON with Padding). Here is how it works and how you can leverage it in your own applications.

Let’s assume I am building an application at labs.loggly.com that will access the API located at loggly.loggly.com. With jQuery, my AJAX call looks as follows:

$.ajax({url: "http://loggly.loggly.com/api/search/?q=ntp", username="guest", password="loggly", ...})

Now, if you do this, you will get the cross-domain error. However, if you just slightly change your call to include an extra parameter, it will succeed:

$.ajax({url: "http://loggly.loggly.com/api/search/?q=ntp",
    username='guest', password='loggly',
    dataType:'jsonp',
    success: function(data) {
        flare = data['data'];
    },
    error: function(XMLHttpRequest, textStatus, errorThrown) {
        alert(textStatus+" - "+errorThrown);
    }
})

Note the newly added dataType parameter. That’s it? Yes, that’s it. It will work like a charm. No more cross-domain security issues. What basically happens are two things. First, the AJAX request that is executed has one more extra query parameter: &callback=?, where the question mark is some string that jQuery randomly generates. The second thing that happens is on the Loggly side. If the callback parameter is present, Loggly does not return the plain JSON element that you would expect, but it wraps it in a function call. Something like:

jsonp12312312({data:{"May-20-2010 12:13:45": 2"}, numFound: 1})

The next thing that happens is that when your browser gets the answer back like this, it will try to execute the function called jsonp12312312. jQuery internally handled that for you by creating a function hook for that function that points to the success function provided to the AJAX call.

That’s really it. We are looking forward seeing your applications that are using the Loggly APIs!

By the way, Loggly is using Django Piston for handling the APIs. The library automatically handles JSONP responses when a parameter called “callback” is present!

Here is how we log from within our application:

from loggly.logging import *

error({'object':'input','action':'create'})

That’s it. The above code creates the following log entry:

Mar 18 15:34:03 app loggly: severity=ERROR,user=logdog_zrlram,request_id=
08BaswoAAQgAADVDG3IAAAAD,object=input,action=create,status=failure

The logging call expects a dict of key-value pairs. This is to enforce key-value based log entries that make it easy for consumers to understand what a specific value means. Without the inclusion of a key, a value is more or less useless. In the example above, note that I only provided two keys: object, and action. However, the log entry contains a number of other data items. Those items are automatically added to the log entries by our logging library without burdening the developer to explicitly include them.

It is probably time to show you Loggly’s logging library:

import logging
import inspect

DEFAULT_LOGGER = 'loggly_web'
logs = None

def logHelper(rest=None, request=None):

    global logs
    output = list()

    # get the logger
    if not logs:
        logs = logging.getLogger(DEFAULT_LOGGER)

    # Loop through all the stack frames until you find the request
    stack = inspect.stack()
    for frame in stack:
        if frame[0].f_locals.has_key('request'):
            request = frame[0].f_locals['request']
            if request is None:
                continue
            # there is a request object
            if hasattr(request,'user') and hasattr(request.user,'username') and len(request.user.username)>0:
                output.append("user="+str(request.user.username).strip())
            if hasattr(request,'META') and request.META.has_key('UNIQUE_ID'):
                output.append("request_id="+str(request.META['UNIQUE_ID']).strip())
            # we found the request object. Get out of here
            break

    # getting input dictionary and appending
    if rest:
        for key in rest:
            output.append("%s=%s" % (str(key.strip()), str(rest[key]).strip()))

    ret = ",".join(map(str, output))
    return ret

def info(rest=None, user=None):

    msg = logHelper(rest, request)
    logs.info(msg)

def error(rest=None, user=None):

    msg = logHelper(rest, request)
    logs.error(msg)

Note that this is only an extract. Download the entire library if you want to use it in your own code. Here are some important things the code does:

• line 17 to 29: This part of the code inspects the call stack to check whether there is an HTTP request object somewhere. The request object contains the username for the session and that is what we automatically extract . This frees the user from manually adding that information to the logging call. Automation is good!
• line 26 and 27: We are using UNIQUE_IDs in Apache. In order to track a request from the Apache logs down into our application, we include that same ID into our Django logs. This is a huge win for associating Apache logs with our application logs.
• line 32 to 24: All the dict entries are added as ‘key=value’ pairs to the log entry. So you can log any key you want.
• line 39 to 47: These are the calls that you use in your code. Note that you can add a user field, which overwrites the username from the request. In some cases that is necessary and useful.

Let us know if you are using our library. I would love to hear back from you. I will post another blog entry later, where I will be talking about how to patch Django itself to do some more logging. We will be looking at how the authentication methods can be extended.

The links:
Django 1.2 Logging Patch
Loggly Logging Library

tar -xzf Django-1.2.tar.gz
cd Django-1.2
patch -p0 < /tmp/django_logging_1.2.patch
python ./setup.py install

This will include the patch into your Django distro and install it. As a next step, you configure your application for logging by adding a snippet similar to the following in your settings.py file:

LOGGING = {
    'loggly_feature': {
        'handler': 'logging.handlers.SysLogHandler',
        'address': ('localhost', 514),
        'facility': 'local3',
        'level': 'INFO',
        'format': 'loggly: %(message)s',        # does not include the severity. it's features, baby!
    },
}

From there you use the following code snippet to log out of your application:

import logging
logs = logging.getLogger('loggly_feature')
logs.error("message")

More information about how to use the logging libraries you find in the Python documentation. It also seems like Django 1.3 will have logging built in. From what I can tell, it looks similar to Simon's approach, but it's not quite the same. I'll keep you posted here!

In my next blog post I will show you how we implemented a logging library for Loggly so that we can very easily log from anywhere within our application.

Django 1.2 Logging Patch

My relationship with Trinity goes back over a year and a half ago – well before Raffy and I started thinking about doing a cloud based log management offering. Like many other startups, Trinity was started by two entrepreneurs. For years, Trinity’s motto has been focusing on early stage companies in specific technology categories, such as cloud computing and systems management.

Loggly is extremely fortunate to be working with Trinity and their bright team, and we greatly value the market experience they bring to the relationship.

History Lessons

I met Trinity through a fairly short introduction path. My good friend and former colleague, Dakota Sullivan, introduced me to a gentleman named Matt Strand in January of 2009. Matt and I had coffee at Crossroads Cafe in South Beach where I told him I was looking to join or start a cloud computing based company. Matt figured he should hook me up with a VC buddy of his, Dan Scholnick at Trinity.

Here’s the email introducing the two of us:

Matt was right about it being a valuable connection. Over the next year Dan and I would spend time together drinking coffee, chatting on the phone, and emailing each other about ideas in and around the enterprise and cloud computing space.

It was because of my conversations with Dan that Raffy and I were able to come to the idea of a cloud based logging service. Even when it came time to start pitching Loggly to others, Dan and Noel assisted us in honing our pitch, which eventually led to us being funded by True in a seed round.

Start Small, Go Big

When you are starting out, even the smallest conversation or the shortest email could potentially be the most important one you’ve had in years. Having an idea, growing it, and turning it into a business is a complicated process. That process takes time, and doesn’t happen over a matter of days, or even weeks, but instead over months and even years.

Our relationship with Trinity has been a long time in the works. While it may have appeared to happen rather quickly, Loggly’s efforts with Trinity started at the very beginning of its life.

In as much as your idea should evolve over time, your ability to convey the idea and the opportunity it represents should grow as well. I’ve lost count of how many times Dan has told me to ‘crisp up’ my presentation, discussed with me partnership negotiation strategies, or told me how to approach feedback with our current private beta testers, but I’m sure the hell glad he did.

Without investors like Trinity and True, it’s unlikely I’d be here telling you this story. You would be well to seek out these types of investors when you are looking for direction and guidance for your idea.

Now you’ll excuse me while we get back to coding. We have beta testers who have logs in need of indexing!

“Despite the economies of scale offered by public cloud providers, private cloud services will prevail for the foreseeable future while public cloud offerings mature, according to Gartner, Inc. Through 2012, IT organizations will spend more money on private cloud computing investments than on offerings from public cloud providers.”

This statement is a bit like NASA doing a press release announcing the moon is continuing to orbit the earth. Wow! The moon, still here next year? That’s awesome. Of course IT is going to spend more money on virutalization for the next few years. The success of the private cloud can be attributed to the fact virtualization has been around for a good while now, and is finally being pressed into mainstream use behind the firewall. Shoot, I think I was running Wine on some of my Linux boxes back in the mid-90s, which means virtualization has been commercialized for at least 15 years at the least. The idea of virtualizing an OS goes back well into the 60s. Come to think of it, so do I.

The public cloud, specifically IaaS and SaaS, is a grouping of emerging technologies. We’re just now starting to figure out how to wield it correctly for new business models. Poking holes in it at this point is simply rabble rousing by companies who’s business models are threatened by it and people who don’t understand it or have a use for it.

It’s a Complicance

Guy Churchward tries to make some good points in his talk with Dave, but at the end of the day, LogLogic is mainly an appliance vendor, and not only do they have big-time COGS to worry about, they also have to figure out how exactly a cloud customer is going to deploy their box on Amazon’s EC2 service. (Hint: They aren’t.) While you might be able to send logs back out of the cloud to an appliance behind the firewall, it’s unlikely to make economical sense to do so in the long term.

While there is a valid point in calling out cloud concerns, security itself is ALWAYS a concern, regardless of whether you run in the cloud or in your own datacenter. Frankly, with Loggly I’m likely better at storing and securing your logs than you are by yourself in your own data center, mostly due to the fact I’m under pressure by multiple people like you to provide a service which is expected at the outset to be secure. It’s no different than the pressure that Google has on them for securing your email, SalesForce for securing your leads, or Amazon securing your credit card info. We’re all culpable here for the security of your data.

Additionally, not all that cloudy data is created equal. A lot of the companies running in the cloud today are web based app companies, and the data they generate is often times very public in nature and not at all affected by compliance concerns. Do you think some user on Flickr cares if I stole all their comments? What about getting access to all those juicy tweets of mine? Oh wait, those are already in the Library of Congress. Nevermind, false alarm!

When IT Rains IT Pours

Log file data is already one of the largest sets of data on the planet. Logging alone in the public cloud is going to be absolutely staggering over the next few years. These trends are being driven by people switching to SaaS based applications, in turn who’s infrastructure either requires the elastic capabilities only the public cloud can provide, or who’s price point can’t be matched by private cloud offerings.

The elastic nature of these infrastructures means the logs which they generate need to be collected and stored in centralized location before the box that generated them disappears. There are many types of logs which are valuable to a company for understanding their business, and not so valuable for those data-thieving ruffians everyone keeps talking about.

While the security access data or net-flow information from public cloud vendors might alleviate the concerns of some consumers, I think there are much higher value adds to these offerings by being able to power availability and analytics services around a company’s application via a log file storage platform.

While the private cloud may continue to orbit peacefully for the next few years, the use of it for web based services will decay eventually, and it’ll be regulated to the more mundane stuff like storing my dental records and tracking my orders over on RadiatorBarn.com.

BTW, I’m still waiting on my radiator, Burton.

The other day I was reading about the Apache and Atlassian hack. Max wrote a really nice summary of how that attack could have been prevented. One of the points he raised was that they should have used HTTPONLY cookies.

I then realized that we might have the same problem with Loggly. After some traffic dumping of our Web sessions, I realized that Django didn’t support httponly cookies. A quick google search revealed that someone wrote a djangosnippet to add httponly cookies. I had to slightly rewrite it, so here is the code I am using:

class cookie_httponly:
    def process_response(self, request, response):
        scn = settings.SESSION_COOKIE_NAME or 'sessionid'
        if response.cookies.has_key(scn):
            response.cookies[scn]['httponly'] = True
        return response

Don’t forget to add the middleware right before the SessionMiddleware. If you are using Python 2.6 or higher, you are done. Unfortunately, we are running Python 2.5, which does not support the httponly flag on cookies. A quick patch solved that problem as well:

--- /usr/lib/python2.5/Cookie.py   (revision 66233)
+++ /usr/lib/python2.5/Cookie.py   (working copy)
@@ -408,6 +408,9 @@
     # For historical reasons, these attributes are also reserved:
     #   expires
     #
+    # This is an extension from Microsoft:
+    #   httponly
+    #
     # This dictionary provides a mapping from the lowercase
     # variant on the left to the appropriate traditional
     # formatting on the right.
@@ -417,6 +420,7 @@
                    "domain"      : "Domain",
                    "max-age" : "Max-Age",
                    "secure"      : "secure",
+                   "httponly"  : "httponly",
                    "version" : "Version",
                    }

@@ -499,6 +503,8 @@
                 RA("%s=%d" % (self._reserved[K], V))
             elif K == "secure":
                 RA(str(self._reserved[K]))
+            elif K == "httponly":
+                RA(str(self._reserved[K]))
             else:
                 RA("%s=%s" % (self._reserved[K], V))

Loggly is now more secure against XSS attacks!

A short while into writing code for the Loggly interface we decided that we needed some eye candy. Given my background in visualization, I was keen on providing our users with an experience that helps them understand their data in an intuitive way.

Over the last few years I’ve been looking into a ton of visualization libraries for the Web. In the past, if you had asked me what library to use for generating charts on your Web site, I would have said, “Use Flash”. While there are a number of interesting Flash libraries out there, the landscape has shifted significantly in the last year. Everyone is moving to JavaScript. After some research, I opted to use a JavaScript charting library called HighCharts. I tried a bunch of other canvas-based libraries, but let me tell you without hesitation, HighCharts rocks.

I am going to show you how we are using HighCharts and how I implemented zooming to dynamically reload more event data on the fly. With any charting library, if you keep zooming in on a chart, it will not progressively load more detailed data. At detailed zoom levels you end up with a small range of data in your graph. Basically if you view a day’s data first, and then zoom into a specific minute, you would only see one data point.

To start, here’s the JavaScript I use to display a chart:

var parse_date = function(data) {
    var result = [];
    $.each(data, function(key, value) {
        var re = new RegExp(/(\d+)-(\d+)-(\d+)T(\d+):(\d+):(\d+)(?:\.(\d+))?/);
        var date = re.exec(key);
        if (date[7] == undefined) {date[7]=0;}
        var real_date = Date.UTC(date[1], parseInt(date[2])-1,date[3],date[4],date[5],date[6],date[7]);
        result.push([real_date, value]);
    });
    return result;
}

chart = new Highcharts.Chart({
    credits: { enabled: false },
    chart: {
        renderTo: 'activity',
        defaultSeriesType: 'area',
        margin: [10, 20, 40, 55],
        zoomType: "x",
            events: {
                selection: function(event) {
                    // change the time frame to be searched
                    var start = Highcharts.dateFormat('%Y-%m-%dT%H:%M:%SZ', event.xAxis[0].min);
                    var end = Highcharts.dateFormat('%Y-%m-%dT%H:%M:%SZ', event.xAxis[0].max);
                    $.ajax({ type: "GET", url: "http://subdomain.loggly.com/api/search/?" \
                        + "q=inputname:logglyapp&starttime="+start+"&endtime="+end \
                        + "&facets=True&buckets=24",
                        success: function(data) {
                             chart.xAxis[0].setExtremes();
                             chart.series[0].setData(parse_date(data));
                             // fix the reset zoom button
                             $('.highcharts-toolbar').click(resetZoom);
                        },
                        error: function(req, text, error) {
                            $("#err").html("Reload error!");
                        }
                    });
                }
        }
    },
    xAxis: { title: { text: 'Time' }, type: 'datetime' },
    yAxis: { title: { text: '# Events' }, min:0,
        plotLines: [{ value: 0, width: 1, color: '#808080' }]
    },
    tooltip: { formatter: function() {
            return Highcharts.dateFormat('%B %e %Y %H:%M:%S', this.x) + '
'+
            ''+this.y+' Events' }},
    plotOptions: {
        area: {
            dataParser: parse_date,
        }
    },
    series: [{ id: 1, name: 'search',
        dataURL: 'http://subdomain.loggly.com/api/search/'
            + '?q=inputname:logglyapp&facets=True'}],
    title: { text: 'traffic last 24 hours' }
});

var reset_zoom = function() {
    // requery for the original data:
    $.ajax({ type: "GET", url: "http://subdomain.loggly.com/api/search/"
        + "?q=inputname:logglyapp&facets=True",
        success: function(data) {
           chart.toolbar.remove('zoom');
           chart.xAxis[0].setExtremes();
           chart.get(1).setData(parse_date(data));
        },
        error: function(req, text, error) {
            $("#err").html("Loading error!");
        }
    });
}
});

Let’s have a quick look at the code. There are two things I want to communicate here: 1. The code I used to display a HightChart graph and 2. The way I am using Loggly’s APIs to query the data.

I mentioned the special zooming that I implemented. Take a look at lines 20 to 39. This is the function that handles zooming, and it is where I am reloading the more detailed data. I set the new start and end dates (lines 23 and 24) and then I am querying the Loggly API with the new timeframe (lines 25 to 27). Upon success – this is important – I am using the chart.series[0].setData() method to set the new data for the chart. The next line overwrites the default button or a link that lets the user zoom out again (lines 32). Note: because you are implementing your own zoom, the default “reset zoom” button from HighCharts will not work anymore and you have to implement your overwrite it with your own function to reset the chart.

The function dealing with the reset functionality is on lines 59 to 72. It does nothing else than query the Loggly API for the original data (I am passing no time parameters) and setting the data just like the previous call. The other thing you have to do is in lines 64 where you need to remove the HighCharts default “reset zoom” link and reset the extremes (line 65).

Moving on, we’ll briefly discuss the way I’m using the Loggly API. If you’d like to use it, you need an account with us. We are currently in private beta, therefore you will need us to give you access to the beta program in order to do so. Email if you want an account to play around with! Back to the code. Make sure you replace the with your actual subdomain. Now that this is out of the way, you can query the API by simply making a GET request to: /api/search. You pass the q parameter with your query. In my example I am getting all the data from my input with the name logglyapp. To get timeline data, you’ll need to pass the parameter facets=True into the call. This will give you counts for time buckets.

To make everything work together, you need one more piece: the date_parse function. You need this part because the Loggly API returns the data with real human readable timestamps and HighCharts wants UTC encoded timestamps. The function on lines 1 to 11 takes care of converting the time for you. Just copy it.

I hope this was useful. Let us know if you are having trouble with any of this. We are looking forward hearing about your graphing endeavors.

If you look at my del.icio.us feed, you’ll find a bunch more visualization and charting links.

If you are running your Web servers behind a load balancer, you have probably noticed that your logs contain the load balancer’s IP address as the client IP, which is kind of annoying. There is an Apache module called RPAF, which fixes exactly this issue. Once you have it downloaded and installed, you configure it as follows:

RPAFenable On
RPAFsethostname On
RPAFproxy_ips 127.0.0.1 10.0.0.1
RPAFheader X-Forwarded-For

The module works such that it takes the IP address that is being transmitted in the X-FORWARD-FOR header and sticks it into the request as the ClientIP.

The not so nice part is that the RPAFproxy_ips statement lets you define the IP addresses of your load balancer. If you have only a set amount of them, all is fine. But here comes the catch. If you are deployed on Amazon AWS and you are using their load balancer (LB), then this solution is going to frustrate you quite a bit. You will notice that the LB’s IP address changes constantly and you keep adding IP addresses to the configuration statement. After about 10 IP addresses, I got sick of that and I started looking at the source code of RPAF to solve this problem once and for all. Here is what I did:

On line 139 of mod_rpaf-2.0.c, I added a return 1; statement. This will tell the is_in_array() function to always assume that the request is coming from a load balancer, without checking the configured list of IP addresses. The rest of the RPAF code is robust enough to only replace the client ip when an X-FORWARD-FOR header is actually set. After the change, do a make install-2.0 and you are in business.

Happy logging!

What I want to focus on today is our integration with RightScale. At Loggly, we use RightScale to manage our AWS instances. Loggly runs three types of servers. (Well, I am simplifying). We have a proxy tier which receives your log messages. The proxy tier, which is basically a bank of machines, forwards the messages to the indexing back end that runs Solr. The third group of machines are the Web or application servers. When a new proxy box comes online, the RightScale management interface knows about the box. I had to know about thse proxies on the application tier (i.e., within Django) as well. How do you do that?

The first solution would be to have the proxies register with Django, as soon as they get online. What happens though when they go down or are taken offline? Seems complicated to keep track of that. Another solution would be to periodically poll the proxies from Django. Not very nice either.

My solution is much more elegant. RightScale has two features that helped me out. The first one is machine tags. Each proxy server is labeled as such. (See Machine Tagging). Secondly, I am using the RightScale API to figure out how many proxies I have and what their IPs are. (As a side note, the RightScale APIs are in Beta right now. There might be changes or improvements coming down the pipe.)

I struggled for quite a bit with using the RightScale APIs out of Python. Here are some things that I learned the hard way and you might find helpful:

Using the API to query all your machines in a specific deployment:

curl -H 'X-API-VERSION: 1.0' -u [user@domain.com]:[password] \

https://my.rightscale.com/api/acct/[account]/deployments/[deployment_number]

Note how you have to add the extra header to request version 1.0 of the API.

Here is how you get all the machines that have a specific tag. Note the structure of my tag! I set role:proxy=true. You need to use this hierarchical model!

curl -H 'X-API-VERSION: 1.0' -u [user@domain.com]:[password] -d'resource_type=server' \
-d 'tags[]=role:proxy' https://my.rightscale.com/api/acct/[account]/tags/search.js

Want JSON output instead of XML, add “&format=js” at the end of your request!

Now, from the response, you would think you could just use that HREF to query an individual server. Wrong. That doesn’t work. You have to add “/settings” in order to make that work:

curl -H 'X-API-VERSION: 1.0' -u [user@domain.com]:[password] \

https://my.rightscale.com/api/acct/20184/instances/[instance_id]/status

Here is how you set a tag on a server: (Note: If you change the tag in the user interface for a running server, it will not take effect. Only if you start a new server of that type, will the tag be there. Unlike the API call, where you can set a tag on a running machine).

curl -H 'X-API-VERSION: 1.0' -u [user@domain.com]:[password] \
-d 'resource_href=https://my.rightscale.com/api/acct/[account]/servers/[server_id]' \
-d tags[]=role:proxy=true https://my.rightscale.com/api/acct/[account]/tags/set

The part I struggled with most was how to call the API from within Python. Turns out httplib2 expects the Web server to respond slightly different than the RightScale server is. If you are using the following code, you will not be able to connect:

h = httplib2.Http()
h.add_credentials(user,password)
response, content = h.request(url, headers=headers)

httplib2 will connect to the Web server without sending the credentials. Only if the server challenges the client to use auth, it will then send the authentication headers. And this is precisely what RightScale is not doing. Therefore, you have to do the following in order to include the authentication headers in the first request already:

h = httplib2.Http()
import base64
base64string = base64.encodestring('%s:%s' % (user, password))[:-1]
headers['Authorization'] = "Basic %s" % base64string
response, content = h.request(url, headers=headers)

Credentials are an interesting topic. I ended up creating a separate user in the RightScale interface that I am using for the APIs. Don’t be fooled though. These credentials still let that user log into the Web interface. I hope that RightScale will add a capability such that I can have a user that can only use the API.

I hope this helps you getting off the ground a bit quicker when using RightScale. Let me know how it goes. You can also find me on Twitter: @zrlram

Logging

I am attending the RSA conference this week. The first session I attended was the Cloud Security Alliance (CSA) meeting. Reading some of the accompanying material and listening to some of the presentations and panels, I couldn’t help it but notice that the terms auditing and logging were all over.

Here is my attempt for an explanation of this. It seems that one of the reasons for this is the nature of the cloud. Think about it. You are in an environment where you don’t control much. You are in an environment where you cannot trust most of the infrastructure pieces. For example, if you are using AWS like we are doing at Loggly, you should generally not trust your AMIs (the OS images). Now, what do you do if you don’t trust someone? You observe them, you monitor them. That’s exactly what is and needs to happen in the cloud: You don’t trust the service. To mitigate this issue, you are going to monitor the service.

And to make this not just my explanation, here is what some panelists during the CSA meeting said:

“Loss of visibility in the cloud” – Scott Chasin, CTO McAfee SaaS Unit
“Lose control and still maintain accountability” – Ken Biery, Verizon Business.

Is the cloud the killer app for logging? And if that’s the case, how do you manage your logs in the cloud? There are hardly any cloud logging solutions out there. I think you see where I am going with this.