LoggingThe Ultimate Guide

your open-source resource for understanding, analyzing, and troubleshooting system logs

curated byloggly

0

Access and Error Logs

Log Files

An Apache log is a record of the events that have occurred on your Apache web server. Apache stores two kinds of logs:

Access Log

Contains information about requests coming in to the web server. This information can include what pages people are viewing, the success status of requests, and how long the request took to respond. It looks something like this:

10.185.248.71 – – [09/Jan/2015:19:12:06 +0000] 808840 “GET /inventoryService/inventory/purchaseItem?userId=20253471&itemId=23434300 HTTP/1.1” 500 17 “-” “Apache-HttpClient/4.2.6 (java 1.5)”

Error Log

Contains information about errors that the web server encountered when processing requests, such as when files are missing. It looks something like this:

[Thu Mar 13 19:04:13 2014] [error] [client 50.0.134.125] File does not exist: /var/www/favicon.ico

Location

Access and error log files are stored on individual web servers. The exact location of your Apache logs depends on your operating system:

Debian/Ubuntu/Linux Mint
Redhat/Fedora/CentOS Configuration
OpenSUSE

General Configuration

The following general default logging configuration directives are specified in absence of specific virtual host container configuration.

Log Level Directive

This specifies log message severity. Default is “warn.”

LogLevel warn

Table of Level Severities

Severity Description Example
emerg Emergencies — system is unusable “Child cannot open lock file. Exiting”
alert Immediate action required “getpwuid: couldn’t determine user name from uid”
crit Critical conditions “socket: Failed to get a socket, exiting child”
error Error conditions “Premature end of script headers”
warn Warning conditions “child process 1234 did not exit, sending another SIGHUP”
notice Normal but significant condition “httpd: caught SIGBUS, attempting to dump core in …”
info Informational “Server seems busy…”
debug Debug-level messages “opening config file …”
trace1-8 Trace messages “proxy: FTP: … ”

Note regarding a particular level: Levels are listed in order of descending severity. When triggered, a configured log level will log all events of that level or greater.

Log Format

Apache offers a ton of flexibility for what you can log. You can find a full description of the Apache log fields listed here in the Apache log documentation.

%a – RemoteIPOrHost %A – LocalIPOrHost %b or %B – Size %D – RequestTimeUs (microseconds) %h RemoteIPOrHost %k – KeepAliveRequests %l – RemoteLogname %r – Request %>s – HttpStatusCode %t – eventTime %T – RequestTimeSeconds %u – RemoteUser %U – UrlPath %v VirtualHost %X – ConnectionStatus %{Referer}i – Referer %{User-agent}i – UserAgent %{UNIQUE_ID}e – UniqueId %{X-Forwarded-For}i – XForwardedFor %{Host}i – Host

You can configure a custom pattern inside your apache configuration file, and then define where you want those logs to be written. Here is an example of one log format you can choose. You read more on the mod_log_config documentation.

LogFormat “%h %l %u %t “%r” %>s %b “%{Referer}i” “%{User-agent}i”” combined CustomLog log/access_log combined

Top 5 Log Fields

Here are some of the most valuable log fields when monitoring server health or for troubleshooting issues. You should consider including each of these in your Apache log format.

  1. HttpStatusCode: This will tell you the exact status of the response. You can read the full list of status codes, but here’s the short version:
    • 2xx is successful
    • 3xx is a redirection
    • 4xx is a client error (a bad request coming from the client or a request that’s not authorized)
    • 5xx is a server error (the request was valid but the server had a problem fulfilling it due to a processing error or timeout of some kind)
  2. UrlPath: Which page the user was trying to load.
  3. RemoteIPOrHost: The IP address making the request. Useful to drill down on traffic from a particular source.
  4. RequestTimeSeconds: How long it took for the request to be processed. This will give you a good idea of how fast or slow your site is. If your site is too slow, you’ll probably need to optimize your servers or your application.
  5. Unique_ID: Also commonly known as a request ID, this helps you search and trace a particular request through to your web application server.

Log Format Notes

These default directives can be thought of as a recipe of formatting assigned to a nickname and used with a CustomLog directive. The format string represents a number of specifiers preceded with a “%” and the specifier character.

A specifier represented as %{Referrer}i means a variable value of type “i,” which in this case means the “Referrer” request header content. The “i” specifies content from the request header.

The “vhost_combined” following the format string, indicated in the example below, is just a name assigned to the format. Apache calls these “nicknames.” Their use with a CustomLog directive is like this: CustomLog <path/to/log> <nickname>. This will log the format represented by the nickname to the specified log location.

Example:

LogFormat “%v:%p %h %l %u %t “%r” %>s %O “%{Referer}i” “%{User-Agent}i”” vhost_combined

Formatting as JSON

Most log management solutions can automatically parse each of the fields out of the Apache log line. If yours does not support this or doesn’t support your desired format, you can also log in JSON. This is a more flexible format that allows you to define your own fields, and it’s also self documenting because the key names describe the data. The main downside is that your logs will consume more volume because you will be including key names with every message. Here is an example for one potential format. You’d insert this in your Apache configuration file as described in the section above.

LogFormat “{ “time”:”%t”, “remoteIP”:”%a”, “host”:”%V”, “request”:”%U”, “query”:”%q”, “method”:”%m”, “status”:”%>s”, “userAgent”:”%{User-agent}i”, “referer”:”%{Referer}i” }”

Once you’ve made the changes to the web server configuration file, you’ll need to restart the Apache web server service.

Combined Access Log Config

Default vhost combined access log config allows for a combined access log for those vhosts without specific location config. Change this config if a new location is desired.

Example:

CustomLog ${APACHE_LOG_DIR}/other_vhosts_access.log vhost_combined

The “vhost_combined” mentioned above is a label or name for a specific format.

Default Virtual Host Overrides

Virtual host (vhost) override logging config is specified in the respective distribution tables below. When duplicating these default files for a specific vhost config, change the default log specification to vhost specific logs if desired. Leaving the default specification will log all vhost access and error log entries to the Apache default.

All log directives specified as general Apache config are overridable in a specific vhost configuration file.

Example standard port 80 vhost config for a site example.com:

ServerName example.com ServerAdmin webmaster@example.com DocumentRoot /var/www/example.com LogLevel info ssl:warn ErrorLog /var/www/example.com/logs/error.log CustomLog /var/www/example.com/logs/access.log example

Example SSL port 443 vhost config:

Debian/Ubuntu/Linux Mint Configuration

The following default configuration directives are specified in the absence of specific virtual host container configuration for the Linux distributions listed.

Debian/Ubuntu/Linux Mint

Table of Default Directives

Directive/Setting Config File Path/Value
*SUFFIX /etc/apache2/envvars (see config file for conditional logic)
**APACHE_LOG_DIR /etc/apache2/envvars export APACHE_LOG_DIR=/var/log/apache2$SUFFIX
AccessLog /etc/apache2/sites-available/000-default.conf CustomLog ${APACHE_LOG_DIR}/access.log combined
ErrorLog /etc/apache2/apache2.conf ErrorLog ${APACHE_LOG_DIR}/error.log
LogLevel /etc/apache2/apache2.conf warn
LogFormat /etc/apache2/apache2.conf LogFormat “%v:%p %h %l %u %t “%r” %>s %O “%{Referer}i” “%{User-Agent}i”” vhost_combinedLogFormat “%h %l %u %t “%r” %>s %O “%{Referer}i” “%{User-Agent}i”” combinedLogFormat “%h %l %u %t “%r” %>s %O” commonLogFormat “%{Referer}i -> %U” refererLogFormat “%{User-agent}i” agent
CustomLog /etc/apache2/conf-available/other-vhosts-access-log.conf CustomLog ${APACHE_LOG_DIR}/other_vhosts_access.log vhost_combined.

* Conditioned environment variable. Provides support for multiple Apache server instances.

** Environment variable. Used to dynamically set the initial log path.

Default Virtual Host Overrides

Virtual host config (port 80):

Redhat/Fedora/CentOS Configuration

The following default configuration directives are specified in the absence of specific virtual host container configuration for the Linux distributions listed.

Redhat/Fedora/CentOS

Table of Default Directives

Directive Config File Path/Value
AccessLog /etc/httpd/conf/httpd.conf /var/log/httpd/access_log
ErrorLog /etc/httpd/conf/httpd.conf /var/log/httpd/error_log
LogLevel /etc/httpd/conf/httpd.conf warn
*LogFormat /etc/httpd/conf/httpd.conf LogFormat “%h %l %u %t “%r” %>s %b “%{Referer}i” “%{User-Agent}i”” combinedLogFormat “%h %l %u %t “%r” %>s %b” common
**LogFormat /etc/httpd/conf/httpd.conf LogFormat “%h %l %u %t “%r” %>s %b “%{Referer}i” “%{User-Agent}i” %I %O” combinedio
*CustomLog /etc/httpd/conf/httpd.conf CustomLog “logs/access_log” combined

* Conditioned on loaded log_config_module.

** Conditioned on loaded logio_module.

Virtual Host Config

Provide a <virtualhost></virtualhost> container within the main config /etc/httpd/conf/httpd.conf or a config file specified within the /etc/httpd/conf.d directory, such as vhost.conf, which is then read by default upon server start.

OpenSUSE Configuration

The following default configuration directives are specified in the absence of specific virtual host container configuration for the Linux distributions listed.

OpenSUSE

Table of Default Directives

Directive Config File Path/Value
AccessLog /etc/apache2/sysconfig.d/global.conf /var/log/apache2/access_log
ErrorLog /etc/apache2/httpd.conf /var/log/apache2/error_log
LogLevel /etc/apache2/sysconfig.d/global.conf warn
*LogFormat /etc/apache2/mod_log_config.conf LogFormat “%h %l %u %t “%r” %>s %b” commonLogFormat “%v %h %l %u %t “%r” %>s %b” vhost_commonLogFormat “%{Referer}i -> %U” refererLogFormat “%{User-agent}i” agentLogFormat “%h %l %u %t “%r” %>s %b “%{Referer}i” “%{User-Agent}i”” combinedLogFormat “%v %h %l %u %t “%r” %>s %b “%{Referer}i” “%{User-Agent}i”” vhost_combined
**LogFormat /etc/apache2/mod_log_config.conf LogFormat “%h %l %u %t “%r” %>s %b “%{Referer}i” “%{User-Agent}i” %I %O” combinedio
***LogFormat /etc/apache2/mod_log_config.conf Logformat “%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x “%r” %b” ssl_commonLogformat “%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x “%r” %b “%{Referer}i” “%{User-Agent}i”” ssl_combined
*CustomLog /etc/apache2/sysconfig.d/global.conf CustomLog /var/log/apache2/access_log combined

* Conditioned on loaded log_config_module.

** Conditioned on loaded logio_module.

*** Conditioned on loaded mod_ssl.

Default Virtual Host Overrides

Virtual host config (port 80):

/etc/apache2/default-vhost.conf

Virtual host config (port 443):

/etc/apache2/default-vhost-ssl.conf

Applicable Modules

The Apache web server offers a number of modules that either change the way Apache works or extend its capability.

mod_log_config:

This module is compiled in by default in the Apache base. Directives Include:

BufferedLogs: Buffers log entries in memory before writing to disk. This is an On|Off switch.

CustomLog: Sets the filename and log file format.

LogFormat: Specifies log file format.

TransferLog: Specifies the log file location.

mod_log_debug

Configurable debug logging. This module is experimental and may or may not be provided in a package distribution. If provided, it will likely require enabling. Directives include:

LogMessage: User-defined messages logged to an error log.

Example: adding configurable debug messages to a location tag similar to below.

LogMessage “/path/to/specific/directory has been requested by” ${REMOTE_ADDR}

mod_log_forensic

Allows forensic request logging and analysis. Often included in package distributions but may require enabling. This module supercedes mod_unique_id for forensic purposes.

A “+” sign precedes the unique identifier for an initial request log entry.
A “-” sign indicates subsequent log entries for the same request.
A final log entry for a given request includes a “-” sign preceding the unique identifier. This indicates final forensic request log entry. Directives include:

ForensicLog: Used to log requests for forensic analysis. Each log entry is assigned a unique identifier.
Forensic Log Format: Specifies a unique forensic ID to the beginning of the log string. This is not a directive, but rather information related to the id itself and the significance of the initial id character, which indicates first and subsequent log writes.
ForensicLog: Specifies a filename for the forensic log location relative to the server root. This directive can be specified in the main server config, or within a vhost config.

Example: Initial forensic log entry calling http://localhost using default log format:

+3264:55958cb4:0|GET / HTTP/1.1|Host:localhost|Connection:keep-alive| Cache-Control:max-age=0| Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8| User-Agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.130 Safari/537.36| Accept-Encoding:gzip, deflate, sdch| Accept-Language:en-US,en;q=0.8| If-None-Match:”2cf6-519e8edfca429-gzip”| If-Modified-Since:Thu, 02 Jul 2015 18%3a51%3a39 GMT

Example: Ending forensic log entry from the above request.

-3264:55958cb4:0

Example: Standard port 80 vhost config for a site including a custom forensic log:

ServerName example.com ServerAdmin webmaster@example.com DocumentRoot /var/www/example.com LogLevel info ssl:warn ErrorLog /var/www/example.com/logs/error.log CustomLog /var/www/example.com/logs/access.log example CustomLog /var/www/example.com/logs/forensic.log forensic

Usage notes:
If an ending log entry is not written, which means there is no “-<unique id>” entry, the request did not complete, indicating script failure.
Once module is enabled, adding a %{forensic-id} variable to a regular log format string will include a forensic identifier in other logs, i.e.:

LogFormat “%{forensic-id} %h %l %u %t “%r” %>s %O” common

mod_logio:

Adds three new logging formats to the mod_log_config format specifications, including byte quantities received, sent, and transferred (combination of received and sent quantities). Normally included in the base Apache compile.
LogIOTrackTTFB: Enables time tracking between the initial request read time and the moment the first byte response is sent.

mod_filter: Provides context-sensitive filters to the output chain by registering any number of filter providers. mod_filter is not specific to logging, but allows for extracting specific requests based on the filter provider. Context containers include: main apache config, vhost config, within directory tags, and .htaccess files.

Employing this module allows for filtering requests containing such things as certain injection criteria and which IP address it’s from.

This module is provided by default in many of the package distributions, but may require enabling. For the purposes of logging, the FilterTrace directive posts information to the error log. Directives include:

AddOutputFilterByType: Assigns an output filter to a particular media type.
FilterChain: Configures a filter chain.
FilterDeclare: Declares a smart filter.
FilterProtocol: Causes the mod_filter to handle response headers correctly.
FilterProvider: Registers filter providers.
FilterTrace: Allows for debugging/diagnostic information to an error log prior to provider processing.

Example: Filtering in a vhost context container conditionally on filter and include modules:

#Declare a resource type filter: FilterDeclare xss #Register a provider: FilterProvider xss INCLUDES %{REQUEST_FILENAME}=”(/[<>]+)$” #FilterProvider … #Build the chain: FilterChain xss #Declare a custom log: CustomLog /var/www/log/xss.log xss #Format the log entry: LogFormat “%h %u %t “%r” %>s “%{Referer}i” “%{User-Agent}i”” xss

mod_unique_id:

Constructs an environment variable and a unique identifier for each request. Often included in package distributions but may require enabling. This unique identifier is written to the access log.

This module has been superseded by mod_log_forensic for forensic purposes, but is still supported for others.

Unlike the forensic identifier, the unique identifier is passed to the application handler via the environment variable UNIQUE_ID. This allows application developers to trace a request through the web server to the application server. It can be useful for debugging a request.

Apache spins off child processes to handle requests, and a child instance processes several requests at a time. As a result, it is sometimes desirable to use a unique identifier to identify a single request across multiple server instances and child processes.

Once enabled, the module provides an identifier by default to the application handler.

Example:

UNIQUE_ID: Vaf3en8AAQEAAAtoQlAAAAAA

The identifier is constructed from a 32-bit IP address, 32-bit process ID, 32-bit timestamp coupled to a 16-bit counter for tighter resolution than a single-second, 32-bit thread index. The timestamp component is UTC to prevent issues with daylight saving time adjustments. The application handler should treat the identifier as an opaque token only and not dissected into constituents.

 

Written & Contributed by

Jason

This guide will help software developers and system administrators become experts at using logs to better run their systems. This is a vendor-neutral, community effort featuring examples from a variety of solutions

Meet Our Contributors Become a contributor