LoggingThe Ultimate Guide

your open-source resource for understanding, analyzing, and troubleshooting system logs

curated byloggly

0

Parsing Apache Logs

Parsing changes a raw text line into individual fields that can be easily analyzed to drill down on specific problems or look at broader trends. This is what a raw Apache log line looks like. It has several fields separated by spaces and quotes.

Let’s consider an example where we want to parse out the status code which is 500. We’ll look at how to do this with simple command line tools or more powerful log management systems.

Using Unix Command Line Tools

You can use Unix command line tools to parse out fields like the status code. Some people prefer to use a tool like grep which can extract data based on regular expressions. The example below extracts patterns of three digits surrounded by spaces. The spaces prevent a match to spurious data such as a timestamp. This will mostly be the status codes, but could include false positives.

Which outputs:

Another example is to use cat, which prints each line, and then use the cut command to take the 9th block of characters that are delimited by spaces. This might have fewer false positives given that you don’t change the Apache log format.

This output looks similar:

Using Log Management Systems

Good log management systems can parse Apache logs, and some can do it automatically while others require configuration. Tools like Logstash or Fluentd require configuration for parsing. They use Grok filters which are regular expressions to extract each field from the log line. Grok also includes a library of regular expressions to extract many formats, but you’ll have the find the right one for you. Here is an example configuration for Logstash:

Services like Loggly can automatically recognize and parse Apache logs. They’ll do it without any configuration if you use one of the common formats. Here is what you’ll see in their expanded event view. Each field has been parsed out and labeled. It’s now ready for the next step, which is analysis!

Written & Contributed by

Jason

This guide will help software developers and system administrators become experts at using logs to better run their systems. This is a vendor-neutral, community effort featuring examples from a variety of solutions

Meet Our Contributors Become a contributor