Behind the Screens


Today we announced the next generation of Loggly. We on the Software Infrastructure are really excited now that our code is publically ingesting, analysing, and indexing customer log data. So what did we use to build our new system? I thought it might be fun to share some high-level details. I’ll discuss each of these in more detail in future blog posts, but let’s stand back and see what’s going on.

High-performance Ingestion…

Right at the front are brand-new, purpose-built Collectors. Written in high-performance C++, Collectors also make significant use of the Boost ASIO framework. Collectors are deployed in a redundant manner, and in multiple locations, ensuring Loggly can always accept your log data.


Downstream of the Collectors is the rock on which we’ve built our ingestion system — Apache Kafka. Kafka allows us to stream incoming data to disk, ensuring we never lose data, and still provide low-latency, high-throughput, service to the rest of the indexing system. We at Loggly really like Kafka — it gives us performance, reliability, scale, and eases Operations significantly. It’s a wonderful example of a technology that does one thing, and does it very, very, well.

…secret sauce…

Much of our secret sauce runs within the next stage, and is built on top of Twitter Storm. In our Storm pipeline we filter, parse, and analyse, log data at high rates. An outstanding feature of Storm is how easily we can add processing resources to this stage, as our requirements — and yours — grow. Our code here is written in Java and Clojure.


The heart-and-soul of our new generation product is ElasticSearch. ElasticSearch, with its intelligent management of large-scale search clusters, allows us to focus on what we are best at — building advanced systems for log search and real-time analytics. And since it has Apache Lucene at its core, we get all the advantages of our in-house Lucene expertise.

…and so much more.

Of course, this is only part of the picture. As part of our new platform we’ve got brand new index-management in place, superior search query handling, timestamp processing, S3 archiving, alerting, and integration with external data sources. It’s been an amazing time to be part of Loggly’s Infrastructure team, and we hope you’ll enjoy the increased reliability, performance, and scale.

Share Your Thoughts