Speedway Motors operates a complex e-commerce site, with millions of pages serving up valuable information and how-to’s, as well as a catalog with tens of thousands of products for auto racing and performance enthusiasts. Speedway’s team of web developers is responsible not only for managing all aspects of the site—interfacing with data sources, integrating third-party APIs, and building presentation logic and user experience—but also for troubleshooting operational problems, 24×7.
“We process a lot of orders through the website every day, both in the United States and internationally,” says Aaron Remaklus, web development manager at Speedway. “Also, we look at the customer experience as an important way we can stand apart from our competitors. With so much money at stake, we can’t afford to not know when an issue could be affecting customers.”
In March 2014, Speedway was preparing to replace its old website with a custom-built, distributed e-commerce platform. The new platform would deliver huge functionality improvements, but the Speedway team also knew that it needed to improve its approach to log management. “One thing we had never done well was log errors,” Remaklus explains. “We did not have a central place to see errors that were occurring or to analyze them. As a result, we sometimes did not become aware of operational issues until our users reported them – a far from ideal situation.”
Speedway was determined to implement a proactive system of monitoring its logs, alerting the on-call web developer about potential issues, and analyzing log data to determine root causes. “Our new website was the most significant project our team has done. We simply would not have been able to go live without having a log management solution in place.”
As the Speedway team prepared to launch the new site, they started a Loggly trial and evaluated the service in its testing environment, using it to debug code and analyze errors that the team intentionally generated during load testing. The team chose Loggly over several other potential solutions because of its:
When the new Speedway Motors site launched in June 2014, Loggly was there to help. All of the site’s internal services log any internal errors or exceptions that occur and send that data to Loggly as JSON. This includes exceptions from third-party services. The logs include a JSON environment field, so that Speedway can aggregate information from all of its environments (development, testing, and production) while limiting alerts to the production site. “The technical aspects of sending logs to Loggly is easy,” says Matt Stubblefield, another member of the Speedway web development team. “The hardest part was figuring out what to log.”
When Speedway first launched the new site, the team took the approach of logging as much data as possible. As they gained experience with the site’s behavior and the capabilities of Loggly, they have made modifications, cutting back logging in certain areas and pushing new information as additional JSON fields.
Loggly has pulled our error reporting out of the dark ages and become an integral component of our e-commerce solution. It’s given us the confidence to release new external customer-facing applications and web services and know that we can react to potential issues instantly as they arise.
— Aaron Remaklus, Web Development Manager
Remaklus and his team set up saved searches that trigger alerts when events reach pre-set thresholds, for example:
All alerts are configured to hit a single PagerDuty endpoint, which notifies the on-call team member. Similarly, Speedway uses AWS CloudWatch to monitor events like spikes in CPU utilization, failed health checks, or auto scaling issues. AWS CloudWatch alerts are also forwarded to PagerDuty.
Speedway team members have one-click access to Loggly within the alert, so they can immediately begin reviewing the relevant logs, determining how many visitors were affected, and finding the root cause. Managing issues in Jira helps them close the loop. “I have never had any problems identifying the source of an issue when I had the logs in Loggly,” Remaklus comments.
The Speedway team has begun to use dashboards as a way to monitor the site and plan future development activities. For example, the team tracks potential unsafe requests. In addition, analyzing bot traffic helped Speedway to surface some old URLs of which they weren’t aware and set appropriate redirects. “We have some platform enhancements coming up, and I think Loggly will be instrumental in helping us plan for them,” Remaklus notes.
With Loggly serving as a catch-all to alert Remaklus and his team to anything that could have an impact on revenue and the customer experience, the new Speedway Motors site has had great success. “I wasn’t expecting to feel so comfortable about being responsible for an application that runs 24x7x365,” Remaklus concludes. “Loggly gives me peace of mind and the ability to sleep at night.”
Speedway Motors is America’s Oldest Speed Shop® and has been a trusted source for specialty rodding and racing products and friendly expert advice for 60 years. Speedway Motors has grown to become the world’s largest manufacturer, distributor and retailer of street rod and racing products.