Alerting with Alert Birds

[obsolete]

Alert Birds is Loggly's alerting and monitoring app. It also serves as the reference app for developers to start working with the Loggly logging platform. We'll get into what that means for app developers in a little bit, but first, let's cover the basics of what Alert Birds does.

You can get started using Alert Birds by navigating to the Alert Birds' website. You'll need a Google Account to manage your Alert Birds app.

There's also an Alert Birds Demo Video demo video you can watch.

Overview

Alert Birds runs your Loggly saved searches and notifies you when something's amiss. There are three major components:

  • Alerts, which run on an interval you select.
  • Saved searches, which are stored on Loggly and are attached to alerts. They're the search query itself, plus the inputs and devices which the search uses.
  • Endpoints, which at this point is really a single type of 'endpoint'. FTM, Alert Birds only uses the PagerDuty system for notification. Multiple PagerDuty endpoints can be defined, however.

 

Saved Searches

The first thing to do is to configure a saved search, which will be stored for you on Loggly using the API. Once you click 'Add Alert', you'll be able to do this. 

 

Endpoints

The next thing to do is to configure a PagerDuty endpoint, so that alerts don't get lost in the shuffle.

If you go to /services/new in your PagerDuty instance (e.g. http://subdomain.pagerduty.com/services/new), you will be able to create a new generic API service that you can use to integrate Alert Birds with Pagerduty:

 

After you click 'Add Service', you'll see your Service API Key on the next page (it will look something like 'c91cffffb0ef012e0d6f48113d009e57'). You'll use that key when adding an endpoint in Alert Birds.

 

So, you just give the endpoint a name, and add some brief text that you want propagated when it's triggered. The alert description, which you'll see next, will contain the details of the alert.

Alerts

After you've configured a saved search and an endpoint, it's time to setup an alert.

 

Alert Birds will run the saved search '500_errors' every 5 minutes, and if there are more than 10 search results, it will trigger an incident in PagerDuty. Once an alert is in a critical state, Alert Birds will re-run the search every minute until the search is no longer hitting the threshold.

The alert description is important, because that will give you context when you get a PagerDuty alert in the middle of the night. The sound that you pick will play if anyone on your Loggly subdomain has a web browser open to Alert Birds when an alert fires. You will also see on-screen notifications:

 

A Couple Important Bits

One caveat with the on-page real-time notifications (and sounds!) is that Pusher and SoundManager2, the libraries we use respectively, need either Flash or HTML5. It's a good idea to unblock Flash if you can for alertbirds.appspot.com, because HTML5 support in SoundManager2 is still in beta. We default to Flash for that library. Don't worry, no matter how much you dislike Adobe and refuse to support their bugware, you'll still get all of your notifications to PagerDuty.

iOS devices require user interaction before they will play sounds, so there's really not much for sound support on iPhones and the like.

Developers

As mentioned above, Alert Birds is intended to be the app that developers can use as a reference when developing against Loggly's APIs.

You can find the Alert Bird code in Loggly's Github repo: https://github.com/loggly/alertbirds-community-edition

We used as many third-party tools as possible to make it as non-Loggly-centric as possible. A partial list of the tools and technologies we used include Google App Engine, Google authentication, OAuth (against loggly.com), Tornado, jQuery, Pusher, PagerDuty, SoundManager2, and WTForms, and it's written in Python and JavaScript. We will be open-sourcing the codebase very shortly on GitHub. In Alert Birds, you'll find useful snippets illustrating a number of common issues, including how you can

  • authenticate against Loggly using OAuth
  • run normal and facet searches
  • create and run saved searches (before we expose them in the Loggly UI!)
  • trigger and resolve alerts in PagerDuty
  • retrieve your input and device lists

The basic idea is for us to make it as easy as possible to create the Loggly app you've always dreamed of (like a CLI shell with vim keybindings!) Please share your thoughts on how we can make the app development process easier.

Heroku Users

Unfortunately we can't integrate Alert Birds with our Heroku addon. However, what you could do is create a syslog drain like this: http://devcenter.heroku.com/articles/logging#syslog_drains that goes to a Loggly account that you sign up for on loggly.com.

– Hoover J. Beaver, Esq.

Top