New Automated Log Parsing: Linux Auth Logs, MySQL Slow Query Logs, Nginx, and Much More!
One of the most important capabilities that sets Loggly apart from other log management vendors is our approach to parsing your log data. Loggly automatically parses, catalogs, and analyzes your data when we receive it. Other cloud-based vendors require you to configure parsing yourself and do parsing at search time, which makes your search results slower. Loggly gives you faster results with Dynamic Fields™, which:
- Visually expose what’s common and what’s unusual
- Make your logs much faster and easier to read
- Offer one-click drill-down to the source of problems
At Loggly, we’re always looking for ways to improve our automated parsing capabilities so that more of our customers’ logs can benefit from Field Explorer and other Loggly features. In this post, I’ll give updates on some changes we have made over the past several weeks.
New Log Types
Linux Authentication Logs
Loggly now extracts fields from the Pluggable Authentication Module (PAM). This includes login information from ssh, sudo, and more. You can use this data to see who might be scanning or trying to hack into your system, and set alerts for key events. Loggly extracts user IDs, IP addresses, and login failure messages. They are available under the “system” log type in Field Explorer.
For example, with this new capability you can see which IP addresses are generating the most failed root login attempts. If these IPs are not part of your organization, you can block them in your firewall or security groups.
MySQL Slow Query Logs
For MySQL, Loggly has added slow query log parsing. With Field Explorer and alerting, we now give you more timely, actionable visibility into how many rows were examined, how long it took to process queries, and whether queries were blocked. You can use these slow query log values to plot time-series charts with Loggly’s Trends feature so you can quickly see changes over time. In addition, MySQL slow query log parsing will allow you to use the numerics range search capability to look for (and alert on) specific ranges of values, for example to trigger an alert whenever a query takes over one second.
Improved Log Types
Apache and Nginx
Loggly’s automated parsing now has support for expanded Apache log formats. You can choose from a variety of formats; there is a list in our documentation. Since Nginx has gained popularity recently—especially with large sites—we have also added support for new Nginx log types in Field Explorer.
For example, you can now quickly generate a pie chart of your Apache status codes. If you have too many 500 errors, you can drill down in Field Explorer to troubleshoot the problem. Additionally, while we currently parse Apache response times, we will soon deploy an update allowing us to parse Nginx response times as well.
If you use Loggly with Heroku, be sure to check out our updated documentation on Heroku logs. Using syslog for log transmission, Loggly is able to extract the name of the component such as web dyno, worker, or something else. We also extract the name of the dyno.
Loggly extracts the Mongo module and timestamp from MongoDB logs. (For more details on the use cases, read Hector Angulo’s blog post from last year.) We have now updated the timestamp field so that it supports two common timestamp formats.
If you’re sending PHP logs via syslog, Loggly parses the method, level, timestamp and message from PHP logs.
If you’re interested in doing custom parsing with Loggly, we have added some useful links to our documentation page. You’ll have step-by-step instructions on how to implement custom parsing using Fluentd, Logstash, Rsyslog, and Syslog-ng.
Bookmark the Automated Parsing Documentation Page
If you haven’t done so already, you should bookmark the Loggly automated parsing documentation page since it’s a handy reference for what Loggly parses. Even if you’re just starting out with Loggly, you’ll get more out of your logs by understanding this important (and differentiating) Loggly capability.