For a game developer like Rumble Entertainment, generating revenue depends directly on two things: its games being up and all of the important features performing well. With three to four releases per week, complex in-game mechanics, and a huge diversity of client hardware and network environments employed by players located throughout the world, this is no small task for Rumble’s team of 40 developers and DevOps professionals. “Our games consume our players’ valuable and limited entertainment time,” says Albert Ho, Executive Producer/Product Manager for Platform at Rumble. “If anything isn’t behaving correctly in our game or if key features are running too slowly, we could lose that customer forever.”
Before Loggly, Rumble built its own log management collector and analysis tool. The project was enormously expensive in terms of developer resources and infrastructure costs, and the service provided only partial visibility into Rumble’s application performance. “Eventually we realized that log management was the poster child of ‘undifferentiated heavy lifting,’” Albert reports. “Getting good at log management wasn’t going to make us better game developers, so we were better off finding a service to take it off our developers’ hands and focusing on our core competency.”
Rumble investigated several cloud-based log management services and quickly settled on Loggly. “We realized that we could double our capacity and triple our data retention time while reducing our monthly costs by more than 75 percent,” Albert comments.
Albert cites the the following Loggly advantages:
Using Log4J and Loggly’s simple REST API, Rumble sends its logs without an agent. The company’s developers were able to get the solution up and running within a day. All of Rumble’s logs adhere to a common JSON structure to facilitate the richest analysis and search so that Rumble can quickly troubleshoot the player experience.
Log management is critical for running an operational service like a game. If we didn’t have Loggly, there would be days where our revenue would be impacted by as much as 70 percent.
— Albert Ho, Executive Producer / Product Manager for Platform
Rumble sends all stack traces and error exceptions to Loggly. The company uses this data to track a set of key metrics that serve as indicators of game health such as error counts and response times. The company also collects performance data (e.g., lags) for a sample of its players to aid in finding more difficult-to-find problems such as slow performance. Finally, it logs platform problems such as failed transactions or external vendor exceptions.
With every release, Loggly enables Rumble to compare post-release and pre-release metrics; as a result, Rumble can often detect potential problems within minutes. If there are any material changes in the key error metrics, the development team uses Loggly to isolate the cause. Albert remarks, “We can usually push a hot x within one to two hours. More traditional game development practices may rely on day-old revenue figures, use ‘black-box’ testers, or worse, rely on their community to let them know about problems.”
“Before Loggly, we might not find out about problems, period… or until a groundswell of player complaints on Facebook sent us into firefighting mode,” Albert says. “Then, it might take us several days of hunting logs on local servers and writing Unix shell scripts to find the cause. Loggly has cut our debugging time from days to as little as minutes. And now we are the first, not last, to know.”
More importantly, every undetected problem, whether it is a broken feature or slow performance, affects player experience and puts revenue at risk. According to Albert, “If we didn’t have Loggly, there would be days where our revenue would be impacted by as much as 70 percent.”
With Loggly data at its disposal, Rumble has taken the next step in creating an excellent player experience: reaching out directly to players who have been affected by certain problems. Once its developers have traced a problem to its source, Rumble can use Loggly to quickly identify the other players that were affected by the same problem. Its customer service team may then o er those players in-game credits to encourage them to come right back to the game.
Rumble maintains weeks of log data, which allows it to immediately analyze health metrics for the last three to four builds and identify potential areas of process improvement. This data also aids in managing external vendors to their service level agreements (SLAs).
The entire Rumble technical staff—development leads, development managers, product manager, testers, and DevOps—now relies on Loggly to keep Rumble’s games running smoothly. In addition, the company has extended Loggly to the third-party developers developing on the Rumble platform. When new developers start working with Rumble, they receive a Loggly account and a JSON message format to use for all of their logs, so that they can share in the benefits of Loggly.