Loggly by Hoover

Building Great Software...5 Cool Tips For Success!

2012-03-15T16:08:03-07:00

Writing software is hard. Writing it well is even harder. I have been rolling some ideas around in my head on how to make software that is better for the people creating it and the people using it. Software has some distinct (and not-so-distinct) phases that it goes through as you work on it. The first of those phases for developers is the design phase. Here are some ideas on making better software during the design process.

If you start with "you know what would be cool?", kill the idea.

OK, so maybe that's a little harsh, but that's my knee jerk reaction when I hear a "you know what would be cool" idea. Let me rephrase my suggestion; stop and think. You may be wondering what I have against doing cool things. Nothing. There's nothing wrong with doing cool things. The problem is that we developers gravitate toward using a cool piece of technology for the sake of using it rather than finding the best fit for our problem. We often attempt to solve a problem either using a cool new piece of technology in a way that it wasn't intended for, or new language features when an alternative would have been better suited. Redis, mongodb, rabbitmq, Python, Ruby, Clojure, Scala, Node.js are all great technologies that meet various needs in their own ways. Sometimes they may be the best fit for your problem, but not always. Just think twice when you think you're about to do something "cool". Developing software is less about being cool and more about building something that works well.

Aim for simplicity.

Software developers have a knack for building and stacking and inheriting and composing until the original simple solution is unrecognizable in a mound of clever engineering. When you're thinking about the next piece of code that you're going to write, start simple. Don't build something in your mind that is more complex than it has to be. You should always use the simplest thing that works, except for when you shouldn't. Next point.

The simplest solution isn't always best.

Sometimes the simplest thing just won't cut it. Sometimes you need that really weird class hierarchy or a really tricky algorithm or something that really hurts your brain to even think about. Usually this is going to happen on the second iteration of a problem. You should still always start with the simplest thing. If that doesn't seem to hold up, go on to something more complicated, but keep the complexity to a minimum.

Don't build more than you need.

It's often tempting to try to build a solution that handles every contingency or meets the needs a user doesn't even know what they have yet. When you are tempted to design everything into your application, think twice. Don't design what you don't need. If you can come up with a good reason for designing it in, then go for it. Otherwise, don't bother. Build just what you need and no more.

Think long-term.

I've seen so many developers (including myself) focus on getting something working now and neglect foreseeable issues with maintenance and deployment (which are the two major areas that this impacts). When you are thinking about the next piece of code that you are going to write, take the time to make sure you are focused on the long term solution. This may sound like it contradicts my previous idea of doing the simplest thing possible, but I think these ideas can co-exist. Start simple. Get something working. But don't plan on getting it "justworking"; plan on getting it working so that the next generation of developers that work on it will be happy you made the decisions you did rather than having them curse your name.

These aren't new thoughts. I've heard them stated by other experienced developers in various ways. And they're not immutable laws of nature. I think they're good guidelines, but the world won't come crashing down on your ears if you do something different. But I do believe that your experience of designing software will benefit from it. And as a result, your software will be better for you and your customer.

Introducing the Loggly Wood Shop

2012-03-02T09:40:14-08:00

We've been talking about doing a Loggly swag store for a while, and I'm excited to announce it's finally here! Now everybody everywhere has access to our snazzy swag. We have some rocking t-shirts, stickers and beer glasses available for purchase. I'm not just saying this because I work here, when I wear Loggly t-shirts out on weekends people compliment me on them.

We’ve always made a point to produce swag that has lasting appeal. All of us have gone to conferences and collected a bunch of stuff, only to take it back to our room, look at it, and leave it there. We are proud to say that people actually wear our swag out and about, track us down at events specifically to get their hands on it, and tweet at us to ask about how they can get it. We've even been offered money for the shirts off our backs, no joke. We are incredibly lucky that the allure of our brand continues beyond our product. Even folks who’ve never used our product, and never will, love the branding and culture we’ve created and want to be a part of it. The Loggly store was born from this enthusiasm and support, and we’re extremely grateful for it.

We’d like to give a special shout out to our fantastic illustrator Andre Jolicoeur who has done an outstanding job bringing Hoover the beaver and the Loggly brand to life with his creations. We also owe a great deal of praise to Shopify, whose service made it a breeze for us to get our store up and running. The Shopify Textmate bundle made development simple and faster than I had anticipated, and the entire launch process was demonstrated clearly through the Shopify interface.

To celebrate our launch we’re offering 20% off all merchandise this week, redeemable by using the discount code beaverlicious at checkout. Happy shopping!

Partnering with Scalr Gets You a Margarita

2012-02-23T16:12:41-08:00

We’ve all dreamed of doing our job while sipping a margarita on the beach, Loggly and Scalr can get you there...well almost. We love things that scale and automate and all things cloud and that is why we partnered with Scalr. Scalr is an open source cloud management tool that brings automation to web applications. The cloud is all about scalability, growth, and making things easier to manage. Scarl gives you the ability to manage as many servers as you need on different cloud computing services and adjusts load capacity as you have spikes and valleys in your traffic. It’s like a system’s admin machine that never sleeps, doesn’t need energy drinks and coffee, so now your system’s admin can actually can have a life. Loggly comes in and takes care of all the logs, we store them, make them searchable, as well as providing features such as monitoring, troubleshooting and user analytics so nothing is lost and you can figure out what your users are doing. Sit back, relax and see what Scalr can do for you.

DNS Management: automatically created and updated for you
Fault tolerance: servers crash, at the worst times, now the problem is detected and automatically resolved for you
Multi-cloud deployments: no need to commit to just one, infrastructure can be spread out across multiple providers
Integrated SSH and key management

We're super excited to be partnered with such a cool company that's helping web apps scale and making your life easier.

Big Data Uncovered?

2012-02-08T15:14:58-08:00

I recently came across an eWeek article titled 2012: A Cloudy Year for Big Data by Frank Ohlhorst. You could easily say I have a few opinions on the matter of big data! :)

First, I agree with Frank’s first notion that big data is neither big or new. The fact is, I've been saying things like "Dude, that's a ton of data!" since I started notching out the opposite sides of floppies back in the 80s. Remember these?

Ohlhprst quickly follows up his vague handwaving that ‘big data’ a new term with, “For most of its existence, big data has been out of the reach of small and midsize businesses (SMBs) because the storage and processing power needed to make this technology work is too expensive.”

Companies have been doing for years what they need to do better business, regardless of whether or not it’s expensive. In manufacturing, the costs of a small company optimizing on how to efficiently making tons of a cheap product can actually be quite a bit more expensive than a larger company making a few units of a complex product. In the same vein, smaller business may have more complex business optimization processes than larger ones, and require relatively larger amounts of data are required to solve those problems than with larger companies.

I agree with Frank that small business typically don't always have the resources necessary to solve massive scale problems, but again the problems are relative. For example, small software startups don't have project managers where larger ones do, not because they can’t afford them, but because they really don’t need them in a full-time capacity. I think this may be part of why SaaS services have been a huge hit and the term cloud has taken off because of it. SaaS allows companies to tackle a wide variety of problems across the entire business, all the while providing cost effective high tech solutions to solve problems in a way you could never have done before. For the first time in history the quality of business processes is experiencing sustained growth.

“These new cloud-based capabilities are on a growth path and are creating more opportunities for even the smallest of businesses to leverage big data without the traditional expenses of compute farms and massive storage arrays.”

Yes. However, compute farms and storage aren't the main thing that these companies need. They need access to the raw data that contains the data about their business, and the tools to extract the data in which they can take action. Figuring out your company’s problems requires brain power, understanding, data and tools. CPU and disks don't solve complex problems. People do.

It's All About Application Analytics

Ohlhprst also describes big data analytics as being comprised of three primary elements: volumes of unstructured data, processing power and algorithms. However, big data doesn't always imply unstructured data. Log files, or what Loggly calls Big Time Data™ typcially contain a large amount of structure. Dealing with structured data isn't always easy, and if you write software that 'expects' a certain structured format, your analysis can sometimes be broken or flawed if it encounters data that doesn't fit the structure you coded for. One way around this problem is to apply extra meta data to the data set. One technique to solving to this problem is adding a search index to the data, which is the approach Google pioniered and what Splunk and Loggly do for log files or time series event data. By being able to do text search data, and interact with it in realtime, or near real time, the user can optimize on solving the problem

Ohlhorst continues, “For it to be true big data, there has to be lots of it, and most SMBs don’t generate that volume of data internally, which leads them to seek out alternative data sources. Here, the cloud delivers.” Not true. Big data should be defined as an amount of data that a human can not reasonably digest. Generating large amounts of meaningful data is actually a bigger problem. Again, it's understanding the problem you have before you can solve it.

Yup. Ohlhorst explains that throughout 2012, data sets and others can be expected to grow exponentially - “The amount of data being generated globally increases by 40 percent a year", according to the McKinsey Global Institute, a data analytics research firm. True. The access of this data, mostly through the web, generates vast amounts of data as well. Ohlhorst continues that information needs to be organized, sorted and processed- and that takes computing power. Frankly nowadays, CPU is cheap enough that most of these problems can be solved on your laptop. Fast CPUs for crunching 'big data' aren't the problem any more than a search engine's main problem is crawling for data. The real bottleneck is adding meaning to the data that a customer can digest and make actionable.

PaaS/IaaS Accessibility Is a Problem

I’m glad that Ohlhprst recognized that Amazon isn’t the only one in the game in offering private cloud-based big data analytics platforms. He believes that since this technology is designed as a complete platform and not as a service, these platforms are still out of the reach of the SMB market.

Ohlhorst is right that these platforms are out of reach - but not just because they are designed as a complete platform and not as a service. I think it's because SMBs don't know they need it, don't have the data to put on it, and don't have the resources to manage it. There are plenty of hosted solutions out there (see SalesForce and their app marketplace) that provide some serious horsepower to the most important task - managing a company's contacts.

And of course, there had to be a Splunk mention in his article. Splunk sells expensive enterprise software. Their software is often times the most expensive piece of software a company has ever bought. Sounds like Oracle, eh? They aren't converting big data analytics into cloud services; they are simply taking their product and making a slimmed down version into a cloud offering they can generate leads with. Any serious big data customer they land will have to buy that very expensive solution and install it on a bank of computers and then pay people to manage it. SaaS is not what Splunk is taking to the market when they go IPO. It's their hellaciously expensive software licenses.

Big Time Data™. It's in the future of your small business.

Getting Your Product Sticky

2012-01-16T23:22:31-08:00

A few months ago I made an off-the-cuff remark about Loggly. "We're like one of those shitty solar powered calculators. When it gets dark, we forget everything you've typed into us."

That comment wasn't far off the mark. Historically, we haven't provided a whole hell of a lot of features that makes it easy to jump back into where you left off on your last search session. Basically when you logged out of Loggly, or even closed the shell in your browser, we'd forget everything you searched for until that point. It made it extremely difficult to get back to something meaningful the next time you logged in.

We shouldn't be here if we aren't meaningful. We should deliver users a 'punch in the gut' feature that makes a lasting impact. One they don't want to avoid.

Saving Time with Sticky Features

Scaling search for a massive amount of log file data being sent in from thousands of machines has been an overwhelming non-trival problem to solve for us over the past year, and it's been our top priority. Unfortunately us solving scale issues aren't readily obvious to users. Users always expect things on the web to be fast. They could care less how hard a problem it was to solve. They don't think to themselves, "Wow, that's fucking fast!". No. Instead they sit around and mutter things to themselves like "Why the hell doesn't feature X do Y? This thing is wasting my time!".

And there it is laid bare: Don't waste your user's time. It's the most valuable resource they have. Get to the point quick, make it easy to get back to what you were doing next time, and do it all with little fuss and muss.

I say give them sticky features!

Saved Search and More

And so, without further ado, I'm officially announcing one of many-to-come new sticky features: saved search. Saved search provides users a way to write a search query and then preserve the search to run again later. Saved searches can generate facet graphs or they can simply run a regular search across a given time range.

The shell has been reworked to provide context changes to use with the saved search feature. You can now change the date context, or limit the context to certain inputs, and then rerun the search or graph using the red rerun button at the top.

Here's a quick screencast running through some of our new sticky features:

Coming Up Soon

We're continuing to add features that increase stickiness to the product. Next week we'll be releasing a revamped history feature for the shell page, where what you've typed in before in a session will be preserved in your command history, just like it would in a normal shell prompt. We're also adding customized graph selection on the main dashboard, which will allow you to start viewing events that matter most to you by default when you first log in.

All these featuers are leading up to a major revamp of the way we provide value for our user's events. Expect completely customized dashboards for server monitoring, website performance, user analytics, and more soon! If you have a feature you'd like to see us implement, please do drop us a line. We're keen on not wasting your time!

Loggly's Outage for December 19th

2012-04-30T17:28:38-07:00

Sometimes there's just no other way to say "we're down" than just admitting you screwed up and are down. We're in the process of rebuilding the indexes of historic data of our paid customers. This is our largest outage to date, and I'm not at all proud of it.

So What Happened?

Yesterday afternoon all of our machines on Amazon's East region, availability zone 1d, were rebooted by AWS staff for maintenance purposes.

The cause of our failure is what some of you on Twitter are calling "a failure to architect for the cloud". I would refine that a bit to say "a failure to architect for a bunch of guys randomly rebooting 100% of your boxes". We've been told by Amazon they actually had to work hard at rebooting a few of our instances, and one scrappy little box actually survived their reboot wrath.

While some might go on a rant about how 'normal' failures don't affect 100% of your boxes the truth is that any and everything (including an army of reboot monkeys) can be expected to happen to your servers if you wait around long enough. The trick to being good at running a reliable service is to architect around any number of everythings that could happen to your service and build for it.

In this case we didn't build the workaround simply because the system we run - a combination of 0MQ+Solr+Zookeeper+Loggly Special Sauce - makes it extremely challenging to survive a complete failure with more than 1/2 of the cluster missing. With other challenges facing us, we decided to live with the risk.

So, How Do We Make This Right?

Single instances of Loggly's search cluster can't be spread across multiple availability zones or regions due to the amount of data we push around, latencies between the search nodes, and the lack of support in our system for redundant indexes. We've been OK with those limitations in the past simple because we systematically archive data to S3 when it arrives and we are capable of rebuilding indexes on the fly if we lose one or more indexers.

Our primary method to address this will be to start sharding our customers across multiple Loggly deployments. This will prevent further outages to the entire customer base. We've already been investigating other data centers on both dedicated hardware and other cloud-based services.

Finally, we accept full responsibility for the impact to our customers. We will be in touch with our paid customers sometime over the next week to address compensation for this outage.

We welcome feedback below.

Kord Campbell, CEO

Enabling CORS in Django Piston

2011-12-05T15:28:32-08:00

Here at Loggly, one of our goals is to make our API accessible and easy to integrate. By enabling CORS (Cross Origin Resource Sharing) on our API endpoints, we hope more Javascript developers can take advantage of what our product has to offer.

CORS is an addition to the browser security model that allows XHR requests to be made from one domain to another. CORS allows Javascript applications to access resources on domains other than the original document's domain, working around the same-origin policy. While Javascript application developers have crafted techniques like JSONP, Flash proxies, XHR receivers, and server-side proxies to circumvent the same-origin policy, CORS makes these hacks unnecessary.

To take advantage of CORS both the server and the browser need to support the standard. The browser needs to initiate a negotiation with the server and the server must signal to the browser which domains are allowed to make cross-domain requests. Our current API is implemented in Django Piston, an open-source project that enabled us to quickly build a RESTful API on top of Django. Piston does not support CORS out-of-the-box, but it wasn't hard to write some code to enable it and we'd like to show how it was done.

A full explanation of CORS is beyond the scope of this post, but the central idea behind CORS is a negotiation between the browser and server of allowed and disallowed actions. This negotiation is done via HTTP headers. The essential headers are the following:

Origin: Sent by the browser signifying the originating domain.
Access-Control-Allowed-Origin: Sent by the server, listing the origin domains allowed to make requests to the server's domain. Can be a comma-separated list of domains or "*" to allow requests from all domains.
Access-Control-Allow-Methods: Sent by the server, listing the HTTP methods the browser is allowed to use in requests to the server.
Access-Control-Allow-Headers: Sent by the server, listing the HTTP methods the server is willing to accept from the browser.

Essentially, to enable CORS we need to have Django Piston respond to an OPTIONS request with the server-sent headers and send the requisite headers along with responses.

The Resource class is the heart of a Django Piston-built API. The code that injects the headers into responses lives in a subclass of the base Resource class. We've called this class CORSResource:

The CORSResource performs two simple tasks. First, it intercepts any OPTIONS method requests to handle the pre-flight negotiation between the browser and the server. Since OPTIONS requests do not have a response body, an empty HTTPResponse() is returned along with the requisite headers. Second, CORSResource intercepts responses from the Django Piston handlers (where responses are generated) and decorates them with the CORS headers.

To use CORSResource, we simply instantiated our endpoints with the CORSResource sub-class instead of the base Resource class. The change to our API's urls.py file look like this:

We hope this post helps other Django Piston API implementors enable CORS in their own APIs. We're planning to release this implementation in the coming weeks and we're looking forward to see what Javascript developers are going to do with direct access to our API.

Happy hacking!

(image from http://blogs.bournemouth.ac.uk/research/2011/09/01/sharing-your-research-data/)

Java Code Coverage With Cobertura and Jenkins

2011-11-29T14:44:13-08:00

Introduction

Writing software is tough, tools and processes that reduce the difficulty are always welcome. Code Coverage is an interesting developer tool because it tells you how much you don't know. Not having a test for a particular piece of code doesn't mean that it doesn't work, it just means you aren't as sure as you could be that it works. In this article, I will be covering getting the open source, Java Code Coverage tool, Cobertura, working with Ant, Jenkins and Github.

Cobertura

Since getting continuous integration working in a particular language can be complicated, it is a best practice to break the problem down into discreet chunks. Fortunately, Cobertura makes this easy, because the source code comes with an example ant project. You can download a binary distribution here: http://cobertura.sourceforge.net/download.html. Inside of the download will be a relative path ..examples/basic. If you cd into that directory, you can generate a code coverage report on a sample Java project by typing:

ant

If you don't have junit installed, you will get output like the below, on my OS X Lion laptop:

If you run ant clean it will remove the reports directory, which contains, old reports. One tricky bit with running the example, is that the build.xml file is set to look for cobertura, and the cobertura lib directory, in a relative path two directories above.

This can be tricky if you check in only the examples directory into your git repository and don't have cobertura installed properly. What will happen then, is that the examples won't build properly and it will be tricky to figure out. If you find yourself in this situation, one easy hack is to simply place the cobertura jar files, including the dependencies in the lib, inside of your ant home. On Ubuntu linux this is /usr/share/ant/lib.

Jenkins

With the basic Cobertura configuration out of the way, the next thing to do is to create a throwaway git repository, on github or your own server. Next you will want to check in the whole Cobertura binary distribution that you download from their site: http://cobertura.sourceforge.net/, or only check in the examples root, and install Cobertura properly on your build server. Inside of jenkins you will need to do the following things:

1. Point Jenkins at your git repository.

2. Install the Cobertura jenkins plugin: https://wiki.jenkins-ci.org/display/JENKINS/Cobertura+Plugin

3. Configure the reports directory correctly. Because I only checked in the examples directory, my reports configuration looked like this:

**/reports/cobertura-xml/coverage.xml

4. build it.

Here is a screenshot of what it looks like when Jenkins builds the example project with code coverage:

Conclusion

If you were able to follow along at home, and have Jenkins automatically building the example project with code coverage, then the next step is to convert this knowledge to your own code base. One problem I had when getting Cobertura working with my code base was configuring the junit stanza properly to fork. Finally, an even simpler way to get code coverage cooking for your Java code base is to install the Eclipse plugin eCobertura. If you are really stuck getting things to work, this is a nice easy win. I hope you enjoyed the article, and if you have any Cobertura tricks, I would love to hear about them.

Ant: http://ant.apache.org/

Cobertura Plugin Jenkins: https://wiki.jenkins-ci.org/display/JENKINS/Cobertura+Plugin

Jenkins: http://jenkins-ci.org/

Github: https://github.com/

Cobertura: http://cobertura.sourceforge.net/

Measuring Code Coverage With Cobertura: http://www.ibm.com/developerworks/java/library/j-cobertura/

eCobertura: http://ecobertura.johoop.de/

Automated Integration

2011-11-18T16:55:15-08:00

One of the fundamental challenges of distributed coding is deciding what/when to integrate. Sure, that patch your colleague just sent you looks good, but is it actually ready to go into master? At Loggly, we've been feeling our way towards a disciplined integration process. A year ago, our frontend developers were all making commits directly to trunk in a single SVN repo. Once every few weeks, we'd run `svn up` on our servers, and hope for the best. Today our code goes through peer review, unit testing, and static analysis before it even touches our master branch.

Like most projects these days, the process starts on github. Fork. Push a feature branch to your repo. Open a pull request. Go through a couple rounds of discussion and revision. Merge. Every change to our code goes through this process. At first we thought it would slow us down, that we'd want pull requests for the nontrivial code and to just push to master for the easy stuff. After just a few days, we found the pull requests were slowing us down not at all, and that we all enjoyed the greater transparency into our colleagues' work.

Once we merge, the automation kicks in -- our default integration branch is 'proposed', so clicking merge doesn't actually get the code into the master branch. Jenkins polls our 'proposed' branch once a minute, then runs a simple preflight script on the code.

Rather than keep that preflight in a jenkins configuration page, we have it checked into the codebase so that any developer can run it too; this way there's no excuse for breaking the build -- you should have seen it break locally =P

Here's our preflight script. Let's go through it line by line.

DIR="$( cd "$( dirname "$0" )" && pwd )"

APP=$DIR/..

First we figure out where we're running, so that we can find the other scripts distributed with the app.

$DIR/purge_pyc

Next we purge pyc files. This is done because if a user recently switched from a branch which contained files which don't exist in our branch, the pyc files may still be around, and may be found by the interpreter.

$DIR/syncenv

Next we run a script to sync our python virtual environment, and ensure all requirements are present.

$DIR/runtests

Here, of course, we run our unit tests. Each run prints a coverage report, so that as we recover from our testing debt, we can measure our progress.

&& $DIR/lint...

Next, and this is important, we run pylint over the parts of our app that we expect to pass with no warnings. As we clean up our app, we continue to add modules to this list. Pylint does a few useful things for us. It looks for trivial name errors of the kind that could quickly cause code to stacktrace -- using a module without importing it, etc. It also enforces certain kinds of coding discipline. Our functions and modules can't exceed a certain length. The cyclomatic complexity of our functions is limited.

If all of this passes successfully, Jenkins automatically pushes the checked-out commit to master, which is where we base our development. Thus, we're always basing our development on known-vetted code.

If any of it fails, Jenkins still has a couple more tricks to pull. Here's our on-failure script:

This comes in two parts. The first runs a standard-issue git-bisect between origin/proposed and origin/master. Since origin/master has already been vetted by jenkins (that's how it became master), we know there'll be a regression somewhere between the commits. This goes into the session output, and is e-mailed to the relevant committers. Next, we roll the proposed branch back to the already-vetted master branch. Whatever pull request broke the build will have to be re-made from scratch.

I'm Presenting at MongoSV!

2011-11-10T14:36:50-08:00

In May I had the opportunity to speak at the MongoDB conference in SF. I ran through how Loggly uses MongoDB to store statistical data on the data we receive and how customers can use our API calls to render visualizations of their logging data. I also hacked a visualization example of events per device using the smashing protovis.js library, which you can view here.

I’m thorougly excited to announce I will be speaking again at the MongoSV conference in Santa Clara in December. I'm stepping up the presentation this time, and will be showing more details about how we've integrated MongoDB into our infrastructure, and how we use MongoDB to generate our roll up reports. I'll run through how we use these roll up stats to drive our own metrics dashboard and how exposing the data to end users via our REST APIs provides a whole host of cool options for visualizing data.

If you’re in town or live within a reasonable distance, defined as the entire Contentintal United States, be sure to come check out the conference! The Mongo/10gen crew always puts on great events filled with lots of technical talks and a rocking after-party.

See you there! Again.

Ajax This!

2011-11-08T14:46:46-08:00

Today’s blog post is more of a tool than a toy. Lately I’ve been working on a bookmarklet that utilizes Pusher (if you want to learn a little about Pusher, check out my previous blogpost). The bookmarklet I’m working on is really silly, but these technologies have the potential to be used for some really cool apps.

I needed to figure out a way for javascript to manipulate objects on web pages remotely through ajax calls. You can already do this with click events and Pusher by sending the class or ID of any element you click on, but what if there is no class or ID? I also want this to work on any webpage. As a result, I needed to figure out how to build a selector that was as unique as possible. One way to do this is to select using all attributes the object has to offer.

Enter, the custom attribute selector syntax:

This selector will only select objects that have a class value of “whatever” AND a myattribute value of “customattribute”.

Some stack overflow and basic google searching told me I can tease attributes out using a regex(gross). I knew there had to be a better way so I started fiddling with javascript objects I captured on click and tried to figure out how to uniqify my selector string. What I discovered is that every object I clicked on contained a ton of info. The parts that I use to build my unique-ish selector are:

localName

The name of the html tag. in the gist above, this value is customhtml.

attribute

All attribute(s) associated with the object (if any). In the gist above, these attributes are class=”whatever” and myattribute=”customattribute”

parentNode

This is the object that contains the one clicked on. Technically my gist has no parent, but I use the parent node to map out a path from the object I clicked on all the way up to the body of the page. If the object you click on doesn’t have any attributes ( a <p> object for example) you can still select it based on the specific path to that object.

textContent

This is any text that’s in the object. For example <p>writing stuff</p> will have a textContent value of “writing stuff”.

Apart these elements aren’t bad at selecting the object you want, but together they get pretty close to behaving like a this object.

Here is how I stitch these elements together to create my this-ish selector:

There are a few tricks here:

I found that if the object you click on has no attributes the path doesn’t help too much, so if the object I’m building a selector for doesn’t have any attributes I just give it a class with an empty value, and it works really well.
In an attempt to keep my selector as unique as possible I’m also using :contains() to select on specific strings as well as the path. This helps ensure that you’re manipulating the exact object you want.
If you’re clicking on an image tag, I found that contains won’t allow you to select that image, but if we take it out it works, so I added a filter to keep image tag selectors from using :contains(). (although it currently only works in the developer tools javascript console, I haven’t yet figured out why it’s not working for my bookmarklet).

Generally, the resulting selector ends up looking something like this:

While this method doesn’t currently work on every single element of every single webpage, it has worked on most of the pages I have tried out, including the google search page.

Anyone who has every tried to explain, on the phone (so 1991), to a grandparent or non-tech savvy friend/family member how to do certain things on the web should be able to see the power of this. With these selectors it’s now possible to build a bookmarklet that allows two or more people to see what elements are being clicking on. Now describing webpage elements to your grandfather isn’t needed. Hook him up with your Pusher powered bookmarklet and all he has to do is go to a URL, click a bookmark button, and let’s say... click on the text you just highlighted in yellow. On the flip side, you’ll know (almost)exactly what elements on the page he’s clicking on because his clicks will make highlights on the page as well. Super neat!

Obviously this method isn’t perfected, but if you’re interested helping me make it more precise, please fork my project on github.

https://github.com/brainTrain/ajaxThis

PagerDuty, Loggly, and Alert Birds

2011-10-28T16:48:52-07:00

Well, I'm back, and this time I'm here to talk about an awesome product that we use all the time, PagerDuty. We use it internally for our own alerting (as do a number of Fortune 500 companies along with a million other startups), but we've also integrated it into Alert Birds, which is our alerting tool. With Alert Birds, you can configure saved searches that run against Loggly, and you'll run those searches over a period of time that you've selected, and Alert Birds will escalate alerts in PagerDuty. Before you can do any of those things, however, you need to set up the PagerDuty endpoint:

After you've done that, the next thing you'll need to do is to configure a saved search, and then configure the alert that you want to run. The search itself is pretty straightforward, it has a name, a search string e.g.

(this is why it's cool to send us JSON!), and a list of inputs and devices that you choose - you may want to run a particular search on only your web servers, for instance. The interesting bit is the alert itself, which runs a search that you choose, but has a number of options as to what conditions consitute an alert, and what the message should be:

This is where PagerDuty comes in. Although you can send a GET or POST request to an endpoint of your choosing with the alert data, triggering an alert in PagerDuty is far more useful, as they can SMS/email/phone you, and they also handle escalations and reporting. So, in the example above, if my web servers are spewing 500 exceptions, I want my ops folks to get notified, provided there are more than 10 - I don't want to wake anyone up over a little blip! I'm a nice IT manager like that. Anyhow, once an alert is in a critical state, it will run your search every minute until you're below the threshold, and once that happens, Alert Birds will automatically resolve your alert in PagerDuty.

That's pretty much all there is to it! You can find the docs on Alert Birds here, please do drop me a line at support@loggly.com if you need a hand, and until next time, happy alerting!

Alert Birds, OAuth, and Loggly

2011-10-10T17:15:15-07:00

In today's edition of the Loggly Blog, we're going to talk about integrating with Loggly using OAuth. You can authenticate against Loggly in a couple different ways, but we're going to focus on OAuth today, because that's how Alert Birds is able to run saved searches on a user's behalf. OAuth can be a little tricky to set up at first, which is a big reason why we're open-sourced Alert Birds, and the code can be found at http://github.com/loggly/alertbirds-community-edition.

To build an app, the very first thing you'll need to do is to register an app in your Loggly instance, at /account/applications/. Apps are called 'Consumers' in OAuth parlance, and after your application gets approved, you can get started working with our API using OAuth.

In basic terms, the OAuth flow looks something like this:

1. Get a request token, which is good for a single request

2. Hit the authorization URL, and pass the request token

3. Log into Loggly when redirected

4. Authorize the application with your consumer key to access Loggly on your behalf

5. Get redirected back to your app, and with the OAuth verifier and the request token, get an access token, which is what you'll need to make future requests

This sounds pretty simple, but everything has to be just right for the flow to work. Query string parameters have to be encoded in the right order, you need to send the correct HTTP method that Loggly is expecting, and you can't lose the OAuth verifier or the request token, because if the access token request fails, you'll have to ask the user to re-authenticate your app - no good!

The OAuth wrapper you'll probably want to use is here: https://github.com/loggly/alertbirds-community-edition/blob/master/lib/oauth.py, and a working client code implementation is here: http://github.com/loggly/alertbirds-community-edition/blob/master/controllers/main.py. The wrapper class simply wraps the OAuth 2 Python library, and makes it a little easier to interact with Loggly, and the client code is where you'll find the interesting stuff in terms of hanging onto the request token, handling the return from Loggly, etc.

A simplified walkthough of what's happening in main.py is this:

That's pretty much all there is to it. Using the lib/oauth.py wrapper from Alert Birds should make it considerably easier to get going. Happy coding!

Get Real with Pusher! Getting Those Alert Birds Squawking...

2011-10-04T21:43:50-07:00

Pusher rocks! It gets real with the web in a major way. Websockets! I'm the new kid on the Loggly block and I've been working on Loggly's first alerting app, Alert Birds. Alert Birds uses Pusher because it's fast (real time is kinda hard to beat) and it's easy to use.

I'm giving you guys some code to play around with. Using Pusher and Sound Manager, we're going to set up a site that makes Loggly's Alert Birds hop and sing on a page whenever anyone clicks on them. That means you see when your buddies click birds, and vise verse. Just think of it as a bird chatroom. Trust me, it's awesome.

If you want to follow along with the code, clone the github repo I created here:

http://github.com/brainTrain/PusherBirds

All you need to do in order to get pusher working is sign up for an account, grab some credentials and get coding.

Pusher has two main components:

Listening for events.
Triggering events.

Let's start with listening:

To listen for a pusher event you only need the app key, channel name and event name. With these compoenets listening to a Pusher channel is pretty straight forward and looks something like this:

Triggering is just as straight forward and requires the same elements as listening, plus the app secret and a backend to handle and send data to Pusher.

First you'll need some data to send, let's go nuts with ajax!

This brings us to the backend component of triggering events, so let's look at the trigger.php file.

The first thing you need to do is head over to pusher's Publisher Libraries and choose the backend language you heart the most.

You can find Pusher's list of Publisher Libraries here: http://pusher.com/docs/rest_libraries

Once you do that simply follow the Library authors instructions and go nuts! In my case I chose the generic php library to demonstrate this simple example. It's all fairly straight forward but I did run into a namespace issue with my host. To fix it I simply commented out the namespace in Pusher.php.

In trigger.php I add Pusher.php using php's require function, put my keys, secrets etc into variables and use Pusher.php's Pusher() class to validate. The last line triggers the event and grabs data from the button variable I sent to the URL using my ajax call.

I should point out that Pusher is somewhat flash dependent as well. For browsers that don't support websockets, they have a solution. Flash. Which means it's probably a good idea to handle flashblockers for any pusher app you create.

Here's Pusher's list of supported browsers: http://pusher.com/docs/browser_compatibility

That's it!

At this point you should have everything you need to communicate with our Alert Birds. Go nuts!

Creating Heroku Addons: Getting Started

2011-09-27T16:46:08-07:00

Recently the Loggly add-on went live for all Heroku users. Creating this add-on involved implementing a few API endpoints in our service for Heroku to call. This is a quick overview of what the development process looks like.

Heroku's add-on API specs have a number of calls that fall into the following three categories:

Provisioning/deprovisioning: These are calls that ask the add-on service to create or remove resources on behalf of the Heroku user. One thing to remember here is that Heroku will ask the add-on service to create resources per Heroku *app*, not per Heroku user account.
In most cases, when Heroku makes a provisioning call to the add-on service, a new account is created for a Heroku app.
Single-sign-on: Heroku users do not need to provide separate account credentials to access services the add-on provides. As long as they are logged into Heroku, they are able to use or configure the add-on. This is accomplished by forwarding the user to the add-on's URL. A token and a timestamp are forwarded along with this request for authentication.
Plan change: Heroku allows add-on providers to change users based on tiers of service. A Heroku user is free to upgrade or downgrade add-on tiers at any time.

Each of these required actions corresponds to a REST endpoint that the add-on service

must implement:

POST and DELETE to http://<addon_service>/heroku/resources respectively provision and deprovision accounts.
GET to http://<addon_service>/heroku/resources/<id>?token=<token>&timestamp=<timestamp> does authentication and any necessary forwarding for SSO.
UPDATE to http://<addon_service>/heroku/resources/<id> will update the account's service tier.

Heroku provides an awesome tool called kensa that simulates API calls from Heroku on these endpoints to test the add-on service.

To install kensa open up a terminal and make sure you have Ruby and gems installed and type:

sudo gem install kensa

Kensa is used to generate the boilerplate manifest that describes properties of the add-on:


Brasero:blogpost ivan$ kensa init

Initialized new addon manifest in addon-manifest.json

Brasero:blogpost ivan$ cat addon-manifest.json 
{
  "id": "myaddon",
  "api": {
    "config_vars": [ "MYADDON_URL" ],
    "password": "QQMmzfm3pkzzE3Fn",
    "sso_salt": "eqs5xOh7kGqen5hd",
    "production": "https://yourapp.com/",
    "test": "http://localhost:4567/"
  }
}

Edit the id to name the add-on. Edit the test and production urls to reflect the hostnames of your respective test and development hosts.

Heroku doesn't call it as such, but kensa is also a tool that cleverly puts test-driven development into your add-on development process. When you run the tool in test mode, it makes REST calls to the host defined in the test directive of the addon-manifest.json file:

Brasero:workspace ivan$ kensa test provision

Testing manifest id key
  Check if exists [PASS]
  Check is a string [PASS]
  Check is not blank [PASS]

Testing manifest api key
  Check if exists [PASS]
  Check is a hash [PASS]
  Check contains password [PASS]
  Check contains test url [PASS]
  Check contains production url [PASS]
  Check production url uses SSL [PASS]
  Check contains config_vars array [PASS]
  Check containst at least one config var [PASS]
  Check all config vars are uppercase strings [PASS]

  Check all config vars are prefixed with the addon id [PASS]
  Check deprecated fields [PASS]
done.


Testing POST /heroku/resources
  Check response [FAIL]
    ! expected 200, got 401

The output above is typical from early runs of kensa, but by running the kensa tool early and often the test failures will guide development.

After all the provision tests pass, invoke the next set of tests for deprovisioning by running:

Brasero:workspace ivan$ kensa test deprovision <some account id>

Follow through with the planchange, and sso tests. When all tests pass, the integration is complete!

What remains is to upload addon-manifest.json to your Heroku provider account and to document the add-on for the Heroku Addon Catalog. More information on how to do that is available here.

Other tips:

Until the add-on leaves Beta, the only plan available to test is the test plan. Be sure your add-on has a test plan during development.
Don't go into Beta until the add-on is feature complete. An add-on cannot be taken out of Beta back into Alpha without help from Heroku.