Simple, accessible log management advances DevOps at Creative Market
The high-level view of what’s available as provided by Loggly Dynamic Field Explorer is key. It makes it a much less laborious process to get meaningful data out of our logs.
Jeremy Koerber Senior Systems Engineer, Creative Market
Highlights:
- Reduces overhead costs and stays ahead of operational problems
- Improves operation of bot-blocking software
- Helps create a DevOps environment
Core Company Beliefs: Simple and Accessible to All
Creative Market is passionate about making beautiful design simple and accessible to everyone so that web creatives can spend more time doing what they love. The same can be said of the company’s drive to foster a DevOps mindset among its development and systems engineering teams: It recognizes the value of simple and accessible log management.
Jeremy Koerber, senior systems engineer at Creative Market, describes the service’s architecture as follows: “We look like a lot of other web companies out there running open source operating systems and software and PHP applications with a MySQL backend.”
Creative Market is hosted in AWS and is an all-Linux, Ubuntu operation. The company relies on a number of Amazon Web Services, such as Amazon RDS for MySQL and Amazon ElastiCache for memcache, which it uses for most of its core pieces of architecture for the application. It also runs its own Elasticsearch cluster. Scalr serves as the high-level orchestration engine that controls the cloud infrastructure, including Chef automation and the Amazon API. “Scalr is our single-pane of glass to control everything,” Koerber explains.
Challenge: Reduce Server Footprint and Offload Log Management Overhead
Before seeking out a log management service, the company was running its own ELK stack. “We were doing all this log aggregation and analysis within our own ELK stack running within Amazon EC2,” says Koerber. “It was a lot of overhead in terms of managing that infrastructure. We wanted to reduce our server footprint and offload the work of keeping our log management function operational.”
Because of Creative Market’s familiarity with ELK, it first explored a managed ELK stack, but ultimately the company chose Loggly. According to Koerber, Loggly’s Dynamic Field Explorer™ feature was a big draw. “There’s something similar in ELK, but it didn’t seem to update quickly. It would show fields from a log file you were no longer looking at. We might have been able to fix it, but we just didn’t have the time to mess around. If there’s a solution that already works, as we’ve found with Loggly, and the cost is comparable, then it makes more sense to go with something where we’re not responsible for patching software or rolling out new features,” he says.
Solution
Koerber reports that Loggly was very straightforward to implement and the process of getting data into Loggly was easy right off the bat. “We’re pretty particular,” he admits, explaining how Creative Market uses an Elasticsearch Filebeat application to collect and ship logs to a Logstash cluster that normalizes the log data before shipping it to Loggly. “We ship all of our logs using the Elasticsearch Filebeat application that collects logs on various boxes and ships them to a Logstash cluster. Logstash then normalizes the log data before we ship them to Loggly,” says Koerber. “We ship one-hundred percent of our logs to Loggly in JSON,” he adds. “It’s just really easy.”
“We’ve got logs coming from Apache, NGINX, application code, and syslog messages from the OS,” explains Koerber. “We also send all of our logs from Chef and other components of our automation.”
Easier Troubleshooting and Performance Monitoring
Creative Market uses Loggly for day-to-day troubleshooting for web servers, load balancers, and performance issues related to keeping an application up. “Loggly makes it easy to correlate the load balancer logs with our application logs and Apache web server logs, so we can see what’s going on.”
Loggly alerts play an important role for Koerber and his team by enabling them to proactively monitor and handle existing problems and anticipate future issues. “As much as we can try to monitor from an external objective standpoint with a tool like Datadog to validate whether the service is up, sometimes that’s just not possible. You may have to supplement your monitoring solutions by alerting on data within your log messages,” says Koerber. “Also you don’t just want to know after the fact. Being able to alert on pre-conditions that may indicate an impending outage is critical for us.”
Koerber and his team are heavy users of Slack and take advantage of Loggly’s Slack integration for sending alerts to a Slack channel. “We’re on Slack all day long. All of our automation components and everything we use for monitoring and alerting notify this one Slack channel,” he says.
Koerber reports that he and his team also rely heavily on Loggly Dynamic Field Explorer™. “Dynamic Field Explorer is critical to us since we’re shipping everything in JSON. We can filter on specific fields that are in common between our logs, like client IP, and trace through. It also helps us discover which fields are available. When someone new comes along, he doesn’t have to know what’s in the logs. He can get a feel for what’s in the logs just by looking at Dynamic Field Explorer,” explains Koerber. “It summarizes the data that’s available and makes it easy to work with.”
According to Koerber, having the data summarized in this way can bring to the surface things he and his team weren’t even looking for. “When you’re looking at raw log data, it’s easy to lose track of the bigger picture. The high-level view of what’s available as provided by Loggly Dynamic Field Explorer is key,” he says. “It makes it a much less laborious process to get meaningful data out of your logs.”
Insights from Loggly Help Create a DevOps Mindset
Koerber’s early career began with a sysadmin role in traditional IT and has transitioned over time into a true DevOps role at Creative Market. “The bulk of my job is automation and writing Chef and Ruby code,” he says.
Because Loggly makes it easy to gain meaningful insights from log data, Koerber believes that it encourages a DevOps mindset for the entire team. “The DevOps paradigm aims to give developers easy access and insight into their work while it’s in production so they can improve upon it and help the operations team run it,” he says. “You’ve got to make it easy to digest the information.”
Loggly Improves Operation of Bot Blocking Software
Creative Market uses Distil, a web-app firewall-managed service that blocks malicious traffic (mostly in the form of bots). However, valid users may be blocked by Distil’s user authentication process in some situations. Creative Market relies on Loggly to understand why this happens and mitigate the effects. “Our Distil logs go into Loggly, so we can correlate them with our application logs to determine why users were blocked,” says Koerber.
Loggly Delivers Lower Overhead, Easy Upgrades, and Anywhere Access
Koerber describes the impact of Loggly on Creative Market’s business in this way: “Loggly has been very well received by the whole team, and it has lowered our operational overhead. The invoice amounts are close, but that doesn’t take into account the time my team would have to spend running our own log management solution.”
He adds, “When Loggly comes out with a new feature, it just shows up. We don’t need to worry about dependencies or upgrade agents. And, anyone can log into Loggly from anywhere.”
