On Monday, Loggly closed a $4.2M B round, with Trinity Ventures leading and True Ventures participating. As you may recall from my previous post, True led our initial seed investment, which was closed 5 months ago to the day.

My relationship with Trinity goes back over a year and a half ago – well before Raffy and I started thinking about doing a cloud based log management offering. Like many other startups, Trinity was started by two entrepreneurs. For years, Trinity’s motto has been focusing on early stage companies in specific technology categories, such as cloud computing and systems management.

Loggly is extremely fortunate to be working with Trinity and their bright team, and we greatly value the market experience they bring to the relationship.

History Lessons

I met Trinity through a fairly short introduction path. My good friend and former colleague, Dakota Sullivan, introduced me to a gentleman named Matt Strand in January of 2009. Matt and I had coffee at Crossroads Cafe in South Beach where I told him I was looking to join or start a cloud computing based company. Matt figured he should hook me up with a VC buddy of his, Dan Scholnick at Trinity.

Here’s the email introducing the two of us:

Matt was right about it being a valuable connection. Over the next year Dan and I would spend time together drinking coffee, chatting on the phone, and emailing each other about ideas in and around the enterprise and cloud computing space.

It was because of my conversations with Dan that Raffy and I were able to come to the idea of a cloud based logging service. Even when it came time to start pitching Loggly to others, Dan and Noel assisted us in honing our pitch, which eventually led to us being funded by True in a seed round.

Start Small, Go Big

When you are starting out, even the smallest conversation or the shortest email could potentially be the most important one you’ve had in years. Having an idea, growing it, and turning it into a business is a complicated process. That process takes time, and doesn’t happen over a matter of days, or even weeks, but instead over months and even years.

Our relationship with Trinity has been a long time in the works. While it may have appeared to happen rather quickly, Loggly’s efforts with Trinity started at the very beginning of its life.

In as much as your idea should evolve over time, your ability to convey the idea and the opportunity it represents should grow as well. I’ve lost count of how many times Dan has told me to ‘crisp up’ my presentation, discussed with me partnership negotiation strategies, or told me how to approach feedback with our current private beta testers, but I’m sure the hell glad he did.

Without investors like Trinity and True, it’s unlikely I’d be here telling you this story. You would be well to seek out these types of investors when you are looking for direction and guidance for your idea.

Now you’ll excuse me while we get back to coding. We have beta testers who have logs in need of indexing!

Dave Rosenberg posted an opinion about cloud based logging yesterday on his Software, Interrupted blog. Dave starts out by mentioning Gartner predicted IT would spend more money on private cloud than public cloud through 2012. Here’s the exact quote from Gartner:

“Despite the economies of scale offered by public cloud providers, private cloud services will prevail for the foreseeable future while public cloud offerings mature, according to Gartner, Inc. Through 2012, IT organizations will spend more money on private cloud computing investments than on offerings from public cloud providers.”

This statement is a bit like NASA doing a press release announcing the moon is continuing to orbit the earth. Wow! The moon, still here next year? That’s awesome. Of course IT is going to spend more money on virutalization for the next few years. The success of the private cloud can be attributed to the fact virtualization has been around for a good while now, and is finally being pressed into mainstream use behind the firewall. Shoot, I think I was running Wine on some of my Linux boxes back in the mid-90s, which means virtualization has been commercialized for at least 15 years at the least. The idea of virtualizing an OS goes back well into the 60s. Come to think of it, so do I.

The public cloud, specifically IaaS and SaaS, is a grouping of emerging technologies. We’re just now starting to figure out how to wield it correctly for new business models. Poking holes in it at this point is simply rabble rousing by companies who’s business models are threatened by it and people who don’t understand it or have a use for it.

It’s a Complicance

Guy Churchward tries to make some good points in his talk with Dave, but at the end of the day, LogLogic is mainly an appliance vendor, and not only do they have big-time COGS to worry about, they also have to figure out how exactly a cloud customer is going to deploy their box on Amazon’s EC2 service. (Hint: They aren’t.) While you might be able to send logs back out of the cloud to an appliance behind the firewall, it’s unlikely to make economical sense to do so in the long term.

While there is a valid point in calling out cloud concerns, security itself is ALWAYS a concern, regardless of whether you run in the cloud or in your own datacenter. Frankly, with Loggly I’m likely better at storing and securing your logs than you are by yourself in your own data center, mostly due to the fact I’m under pressure by multiple people like you to provide a service which is expected at the outset to be secure. It’s no different than the pressure that Google has on them for securing your email, SalesForce for securing your leads, or Amazon securing your credit card info. We’re all culpable here for the security of your data.

Additionally, not all that cloudy data is created equal. A lot of the companies running in the cloud today are web based app companies, and the data they generate is often times very public in nature and not at all affected by compliance concerns. Do you think some user on Flickr cares if I stole all their comments? What about getting access to all those juicy tweets of mine? Oh wait, those are already in the Library of Congress. Nevermind, false alarm!

When IT Rains IT Pours

Log file data is already one of the largest sets of data on the planet. Logging alone in the public cloud is going to be absolutely staggering over the next few years. These trends are being driven by people switching to SaaS based applications, in turn who’s infrastructure either requires the elastic capabilities only the public cloud can provide, or who’s price point can’t be matched by private cloud offerings.

The elastic nature of these infrastructures means the logs which they generate need to be collected and stored in centralized location before the box that generated them disappears. There are many types of logs which are valuable to a company for understanding their business, and not so valuable for those data-thieving ruffians everyone keeps talking about.

While the security access data or net-flow information from public cloud vendors might alleviate the concerns of some consumers, I think there are much higher value adds to these offerings by being able to power availability and analytics services around a company’s application via a log file storage platform.

While the private cloud may continue to orbit peacefully for the next few years, the use of it for web based services will decay eventually, and it’ll be regulated to the more mundane stuff like storing my dental records and tracking my orders over on RadiatorBarn.com.

BTW, I’m still waiting on my radiator, Burton.

The other day I was reading about the Apache and Atlassian hack. Max wrote a really nice summary of how that attack could have been prevented. One of the points he raised was that they should have used HTTPONLY cookies.

I then realized that we might have the same problem with Loggly. After some traffic dumping of our Web sessions, I realized that Django didn’t support httponly cookies. A quick google search revealed that someone wrote a djangosnippet to add httponly cookies. I had to slightly rewrite it, so here is the code I am using:

class cookie_httponly:
    def process_response(self, request, response):
        scn = settings.SESSION_COOKIE_NAME or 'sessionid'
        if response.cookies.has_key(scn):
            response.cookies[scn]['httponly'] = True
        return response

Don’t forget to add the middleware right before the SessionMiddleware. If you are using Python 2.6 or higher, you are done. Unfortunately, we are running Python 2.5, which does not support the httponly flag on cookies. A quick patch solved that problem as well:

--- /usr/lib/python2.5/Cookie.py   (revision 66233)
+++ /usr/lib/python2.5/Cookie.py   (working copy)
@@ -408,6 +408,9 @@
     # For historical reasons, these attributes are also reserved:
     #   expires
     #
+    # This is an extension from Microsoft:
+    #   httponly
+    #
     # This dictionary provides a mapping from the lowercase
     # variant on the left to the appropriate traditional
     # formatting on the right.
@@ -417,6 +420,7 @@
                    "domain"      : "Domain",
                    "max-age" : "Max-Age",
                    "secure"      : "secure",
+                   "httponly"  : "httponly",
                    "version" : "Version",
                    }

@@ -499,6 +503,8 @@
                 RA("%s=%d" % (self._reserved[K], V))
             elif K == "secure":
                 RA(str(self._reserved[K]))
+            elif K == "httponly":
+                RA(str(self._reserved[K]))
             else:
                 RA("%s=%s" % (self._reserved[K], V))

Loggly is now more secure against XSS attacks!

A short while into writing code for the Loggly interface we decided that we needed some eye candy. Given my background in visualization, I was keen on providing our users with an experience that helps them understand their data in an intuitive way.

Over the last few years I’ve been looking into a ton of visualization libraries for the Web. In the past, if you had asked me what library to use for generating charts on your Web site, I would have said, “Use Flash”. While there are a number of interesting Flash libraries out there, the landscape has shifted significantly in the last year. Everyone is moving to JavaScript. After some research, I opted to use a JavaScript charting library called HighCharts. I tried a bunch of other canvas-based libraries, but let me tell you without hesitation, HighCharts rocks.

I am going to show you how we are using HighCharts and how I implemented zooming to dynamically reload more event data on the fly. With any charting library, if you keep zooming in on a chart, it will not progressively load more detailed data. At detailed zoom levels you end up with a small range of data in your graph. Basically if you view a day’s data first, and then zoom into a specific minute, you would only see one data point.

To start, here’s the JavaScript I use to display a chart:

var parse_date = function(data) {
    var result = [];
    $.each(data, function(key, value) {
        var re = new RegExp(/(\d+)-(\d+)-(\d+)T(\d+):(\d+):(\d+)(?:\.(\d+))?/);
        var date = re.exec(key);
        if (date[7] == undefined) {date[7]=0;}
        var real_date = Date.UTC(date[1], parseInt(date[2])-1,date[3],date[4],date[5],date[6],date[7]);
        result.push([real_date, value]);
    });
    return result;
}

chart = new Highcharts.Chart({
    credits: { enabled: false },
    chart: {
        renderTo: 'activity',
        defaultSeriesType: 'area',
        margin: [10, 20, 40, 55],
        zoomType: "x",
            events: {
                selection: function(event) {
                    // change the time frame to be searched
                    var start = Highcharts.dateFormat('%Y-%m-%dT%H:%M:%SZ', event.xAxis[0].min);
                    var end = Highcharts.dateFormat('%Y-%m-%dT%H:%M:%SZ', event.xAxis[0].max);
                    $.ajax({ type: "GET", url: "http://subdomain.loggly.com/api/search/?" \
                        + "q=inputname:logglyapp&starttime="+start+"&endtime="+end \
                        + "&facets=True&buckets=24",
                        success: function(data) {
                             chart.xAxis[0].setExtremes();
                             chart.series[0].setData(parse_date(data));
                             // fix the reset zoom button
                             $('.highcharts-toolbar').click(resetZoom);
                        },
                        error: function(req, text, error) {
                            $("#err").html("Reload error!");
                        }
                    });
                }
        }
    },
    xAxis: { title: { text: 'Time' }, type: 'datetime' },
    yAxis: { title: { text: '# Events' }, min:0,
        plotLines: [{ value: 0, width: 1, color: '#808080' }]
    },
    tooltip: { formatter: function() {
            return Highcharts.dateFormat('%B %e %Y %H:%M:%S', this.x) + '
'+
            ''+this.y+' Events' }},
    plotOptions: {
        area: {
            dataParser: parse_date,
        }
    },
    series: [{ id: 1, name: 'search',
        dataURL: 'http://subdomain.loggly.com/api/search/'
            + '?q=inputname:logglyapp&facets=True'}],
    title: { text: 'traffic last 24 hours' }
});

var reset_zoom = function() {
    // requery for the original data:
    $.ajax({ type: "GET", url: "http://subdomain.loggly.com/api/search/"
        + "?q=inputname:logglyapp&facets=True",
        success: function(data) {
           chart.toolbar.remove('zoom');
           chart.xAxis[0].setExtremes();
           chart.get(1).setData(parse_date(data));
        },
        error: function(req, text, error) {
            $("#err").html("Loading error!");
        }
    });
}
});

Let’s have a quick look at the code. There are two things I want to communicate here: 1. The code I used to display a HightChart graph and 2. The way I am using Loggly’s APIs to query the data.

I mentioned the special zooming that I implemented. Take a look at lines 20 to 39. This is the function that handles zooming, and it is where I am reloading the more detailed data. I set the new start and end dates (lines 23 and 24) and then I am querying the Loggly API with the new timeframe (lines 25 to 27). Upon success – this is important – I am using the chart.series[0].setData() method to set the new data for the chart. The next line overwrites the default button or a link that lets the user zoom out again (lines 32). Note: because you are implementing your own zoom, the default “reset zoom” button from HighCharts will not work anymore and you have to implement your overwrite it with your own function to reset the chart.

The function dealing with the reset functionality is on lines 59 to 72. It does nothing else than query the Loggly API for the original data (I am passing no time parameters) and setting the data just like the previous call. The other thing you have to do is in lines 64 where you need to remove the HighCharts default “reset zoom” link and reset the extremes (line 65).

Moving on, we’ll briefly discuss the way I’m using the Loggly API. If you’d like to use it, you need an account with us. We are currently in private beta, therefore you will need us to give you access to the beta program in order to do so. Email if you want an account to play around with! Back to the code. Make sure you replace the with your actual subdomain. Now that this is out of the way, you can query the API by simply making a GET request to: /api/search. You pass the q parameter with your query. In my example I am getting all the data from my input with the name logglyapp. To get timeline data, you’ll need to pass the parameter facets=True into the call. This will give you counts for time buckets.

To make everything work together, you need one more piece: the date_parse function. You need this part because the Loggly API returns the data with real human readable timestamps and HighCharts wants UTC encoded timestamps. The function on lines 1 to 11 takes care of converting the time for you. Just copy it.

I hope this was useful. Let us know if you are having trouble with any of this. We are looking forward hearing about your graphing endeavors.

If you look at my del.icio.us feed, you’ll find a bunch more visualization and charting links.