LoggingThe Ultimate Guide

your open-source resource for understanding, analyzing, and troubleshooting system logs

curated byloggly

0

Centralizing Java Logs

Java applications can easily distribute logs to multiple destinations. In an enterprise environment, you’ll likely have logs stored on different systems, requiring you to log in to multiple servers. Even still, those logs could have been created using different Appenders and different Layouts. This fragmented approach to logging makes it difficult to retrieve data in order to monitor, troubleshoot, or maintain your application.

Centralized logging resolves these issues by unifying the format and storage of your log data. Log data is sent from your application to a service, where it’s automatically parsed and imported into a database. Many centralized logging systems aren’t limited to programming languages and support output from scripts, services, and system commands. Not only do logging systems consolidate log data from various sources, but they also process and present data through an accessible interface.

This section covers some of the ways you can implement centralized logging in your Java applications. We’ll focus on logging from a framework.

Benefits of Centralizing Java Logs

There are three key benefits to managing logs through a centralized logging system:

  1. Log data is stored in a single location. Instead of having to retrieve logs from multiple systems, you can access all of your logs from a single interface.
  2. Log data is automatically parsed. Logs come in hundreds of different formats including plain text, JSON, XML, and HTML. Logging systems automatically detect the type of log being read and convert it into a standard format.
  3. Log data is searchable, indexable, and exportable. Log entries are broken down into their key components, which can be searched and filtered. Logs can also be archived, exported, or redistributed without having to touch the original log file.

With a centralized logging system, you can have multiple applications on multiple systems storing logs in a single location, saving you time and effort.

Popular Tools to Centralize Logs

There are many popular tools to centralize logs, and your choice of which ones to use depends on your needs.

Java Logging Frameworks

If you want to log directly from your Java application code, then you can use a logging framework such as Log4j, Logback, or SLF4J. You can forward logs to the server logging daemon or directly to a log management service (see below).

Server Logging Daemons

You can forward system or application logs through your server logging daemon. Popular ones include rsyslog, syslog-ng, and nxlog. It can store the logs locally or forward them on to a log management solution.

On-Premise Log Management Software

Centralized logging software lets you deploy your own self-managed log management solution. These can range from simple collectors to complete web applications. Examples include Splunk, Logstash, Graylog, Fluentd, and Kafka.

Cloud-Based Log Management Services

Cloud-based logging services remove the burden of maintaining your own logging solution. Log entries are forwarded to a third party where they can be accessed via a web browser or app. Examples include Splunk Cloud, Loggly, Sumo Logic, Papertrail, and Logentries.

Output Methods

Once you’ve chosen a logging system, the next step is to ensure your logs are being delivered. Different Appenders provide different methods for transmitting logs from your application to your logging system.

File

One of the most common log destinations is a file. Events collected by the logging framework are written to a location on a local disk. The main advantage of files compared with the console is that the log events persist after the application exits or the console closes. Assuming the log output is plain text, log files can easily be reviewed in a text editor or console for troubleshooting application issues. Some drawbacks include disk I/O latency, potential file permission issues, and the consumption of disk space. To prevent exhausting available memory or disk space, utilities such as logrotate can split a single log file into multiple files once the file reaches a certain size or according to a regular schedule.

One major disadvantage of file logging is that it can be difficult to maintain log events that span multiple lines, such as a stack trace. They are easy to read in a text editor, but often a program reading or analyzing the logs will consider each line as a separate event. One alternative is to use a structured Layout that can store events on a single line. With Log4j and Logback, you can store JSON and XML events on a single line by setting the compact attribute to true. Another solution is to monitor the log file using a program that can recognize indented stack traces, which is explained in more detail in the Syslog Stack Traces section.

Syslog

Advanced logging libraries provide the option to output log events to a syslog daemon. The syslog daemon is a widely used logging destination, especially in Unix-based operating systems. The syslog daemon accepts log messages from files, devices, or even remote systems. The main advantage of using a syslog daemon is that it’s a separate process from your application, so it can asynchronously transmit logs without affecting your app’s performance. Additionally, common daemons have internal queues making them resilient to network interruptions. When your syslog daemon sends logs to a central server, it’s important to pick the right transport layer protocol—either UDP or TCP. For message reliability, TCP is the recommended transport protocol, since it’s possible for UDP to drop network packets or deliver them out of order.

One disadvantage of syslog is that it doesn’t support multi-line events. Your logs will need to be structured in a way that stores them on a single line, or import them using a module that automatically converts them.

HTTP/S

Some logging systems support the transmission of log data using HTTP. HTTP and HTTPS are the application layer protocols that drive the worldwide web. HTTP is the default, while HTTPS adds encryption and validation. The benefit of HTTP/S transmission is that it allows for multi-line events and provides built-in security using HTTPS.

There are several HTTP Appenders available for Log4j version 1 and Logback, as well as custom Appenders for different logging services. Many log management services like Loggly offer custom appenders such as the logglylog4j library.

Async versus Sync

Normally, logging calls are synchronous, which means the program can’t continue executing until the log event is recorded and acknowledged. This can result in noticeable overhead, especially when the application logs frequently or when the individual log calls are expensive. For example, socket-based logging can cause latency resulting from network delays or failures. Calls like these are strictly prohibited in Android, which will throw an exception if an application attempts to perform network operations on the main thread.

Alternatively, applications can use the asynchronous model for logging. In this model, logging calls queue requests to record a log event on another thread. This mitigates latencies and offers higher throughput of logging events, especially during logging bursts. One disadvantage of asynchronous logging is the increased complexity of error handling. Instead of checking the result of the logging call in the next line, the caller must handle the error another way, for instance, by providing a callback that retries the request. Another disadvantage is that events might be lost before the program records them, which could be a deal breaker for auditing applications. For example, helpful troubleshooting data could be lost when your application is asynchronously logging an IOException, but crashes before the asynchronous call can be completed. One technique to alleviate this problem is to use synchronous Loggers for ERROR-level events and asynchronous Loggers for everything else. Additionally, you might want to think how your application responds when a network link goes down. Will your threads time out eventually, or will you insert logs into a queue and send them when the link resumes?

In this Logback example, all log events with INFO severity or higher are logged asynchronously to a file, and all ERROR events from the com.example Logger are synchronously sent to syslog.

Centralizing Multi-Line Stack Traces

Multi-line stack traces are often harder to process than regular log entries. Stack traces vary in length and have multiple sections, making them difficult to parse through pattern matching. The parser needs to know where the stack trace begins, where it ends, and how each line relates to the event.

In some cases, each line of the stack trace is treated as a separate event. This is a common problem in plain text logs, which can have a wide variety of layouts and patterns. For more information, see the Parsing Multi-Line Stack Traces section.

HTTP/S

HTTP/S natively supports new lines, so your stack traces can be sent without modification. You’ll have to choose an Appender that supports HTTP/S as a transport protocol. One disadvantage is that the HTTP/S protocol is heavier than syslog because it includes more headers and requires acknowledgement. Also, you’ll need an HTTP/S endpoint or collector ready to receive the logs. Cloud-based services like Loggly provide this service.

Syslog Protocol

Unfortunately, the syslog protocol was written to support a single line per event. By default, Logback and Log4j’s SyslogAppenders log each line of a stack trace with a new line, which is interpreted as a separate event.

One way to overcome this is using an Appender that writes a UDP or TCP packet with the new lines still in them. Rsyslog will recognize and convert  it to a single-line event by replacing  the “nr” characters with the octal codes “#012#011”. You can ignore or reconvert these octals to new lines in your log management solution.

If you are using file monitoring, you should configure your syslog daemon to recognize multi-line events. You can configure rsyslog’s imfile with paragraph read mode. This treats an empty line between each stack trace as an event separator.

For example, this program generates exceptions while trying to load two missing files. Log4j’s PatternLayout includes the %xEx conversion pattern. Adding it to the pattern lets us separate each log entry with an empty line:

These log entries are stored in the /var/log/myLog.log file. The following imfile configuration scans myLog.log every 10 seconds for changes and forwards each event to rsyslog. Note that the latest versions of rsyslog introduced a new configuration syntax, whereas this example uses the legacy syntax.

The InputFileTag parameter specifies a unique tag for entries originating from this file. The InputFileReadMode parameter determines how imfile treats multi-line messages. A value of 1 tells imfile to read in paragraph mode, treating blank lines as message separators. InputFilePollInterval sets the number of seconds between each check of the log file. Lastly, InputRunFileMonitor activates the newly configured monitor. Restart the rsyslog service, and the log data appears in the syslog as complete events with octal characters.

Logging from Android

Android provides a built-in logging API through the android.util.Log class. Android also supports several logging frameworks including Androlog, Timber, SLF4J, Logback, and Log4j.

Android’s built-in logging API prints log data to LogCat, which provides a buffer that stores log data for access over ADB and DDMS. android.util.Log is designed for development and debugging, making it an unfavorable solution for collecting and centralizing log data when using the app in a live environment. Other logging frameworks provide ways to store and submit log data without having to use a separate debugging tool or remote shell.

Most logging frameworks on Android will use HTTP/S as a transport protocol because there is no local syslog daemon to forward logs. They will use a cloud-based endpoint to collect and store the logs.

Logging with Logback

There are two ways to use Logback on Android. SLF4J provides the slf4j-android package, which routes all SLF4J log requests to Android’s Log class. On the other hand, the logback-android package allows you to use Logback as the logging utility on Android. With access to Logback’s API, you can record and forward log events to your destination of choice.

For example, we can add a Logback Appender to our app by importing from org.slf4j and creating a new Logger:

Logback-android will read the configuration file stored in assets/logback.xml:

Logback-android also supports configuration through your app’s AndroidManifest.xml file. For more details, see the logback-android project page.

Logging with Timber

Timber is a logging framework that extends Android’s Log class, adding a lightweight and extensible API. Timber uses static methods to log data, and logging behavior is configured through instances of Tree objects. Trees are initialized at the start of the program and contain the code necessary to forward log entries to their proper destination. In many ways, Trees are similar to Appenders.

This example shows how to log a few simple actions using Timber. We’re using a LogglyTree provided by the timber-loggly library to automatically send log data to a centralized logging system. The resulting logs are automatically formatted as JSON:

Below is a screenshot of the app at startup. A “logged message” toast appears at startup and when the “Settings” menu option is selected. The toast indicates that a message was sent to Loggly. Note that the log event is sent as JSON in this form:

This means that the log event triggered before the toast (Timber.i(“Activity created!”);) will appear in our logs like this:

timber loggly demo

The image below shows that Loggly received the messages from the example app:

loggly example

Removing Logs from Release Builds

Android developers often prefer to remove some or all logging from release builds to improve overall performance and reduce package size. This can be accomplished with ProGuard by setting up rules that strip out the logging calls. For example, one would add the following rules to his or her application’s proguard-rules.txt file in order to remove DEBUG, VERBOSE, or INFO level logs:

There are cases where logging is preferred to be kept in the release builds, in which case you’d want to prevent ProGuard from removing those calls. When using SLF4J and Logback, be sure to include these rules to retain your logging calls:

Logging from Tomcat

Tomcat comes with a robust logging system which allows multiple logging frameworks to work independently of each other. Known as JULI (the Java Utility Logging Implementation), Tomcat’s logging system extends the Apache Commons Logging framework to provide new features and flexibility.

JULI’s strength lies in its ability to separate your application’s logging framework from Tomcat’s logging framework. This makes it possible to use your framework of choice in your web applications—even java.util.logging—without interfering with other applications or with Tomcat. For a full overview of JULI’s features, see Apache’s guide to logging in Tomcat.

Some distributions of Tomcat are simplified to use a hard-coded java.util.logging, although it can be reconfigured to allow for different frameworks including Log4j, Logback, and SLF4J using JULI.

Tomcat Logging Setup and Configuration

Throughout this section, you’ll see many references to “Catalina.” Catalina is Tomcat’s servlet container and handles several tasks including starting and stopping servlets, redirecting requests to servlets, and managing access rights. The following table shows the location of Catalina’s configuration files for various operating systems. By default, configuration files are stored in the $CATALINA_HOME directory. If you’re running multiple instances of Tomcat, then the files will be stored in the $CATALINA_BASE directory.

Distribution $CATALINA_HOME $CATALINA_BASE
Ubuntu / Debian /usr/share/tomcat/ /var/lib/tomcat7/
CentOS / RHEL /usr/share/tomcat/ /var/lib/tomcats/

JULI’s logging behavior is configured through a logging.properties file, which can be set on a global or per-application level. The global configuration is available at $CATALINA_BASE/conf/logging.properties. If the global configuration file is missing or unreadable, then JULI defaults to the logging.properties file used by the system’s Java installation. For application-specific configurations, the configuration is stored in the application’s WEB-INF/classes/ folder.

JULI uses the same configuration syntax as java.util.logging, with a few minor exceptions. Global logging settings are stored in $CATALINA_HOME/conf/logging.properties, while application-specific properties are stored in WEB-INF/classes/logging.properties. You can also configure logging behavior in the code of your application.

For more information on configuring java.util.logging, see the Java Logging Basics section.

Default Output for Tomcat Logs

By default, Tomcat logs to $CATALINA_HOME/logs/catalina.out. All System.out and System.err logs are redirected to this file, as well as any uncaught exceptions. While Tomcat itself doesn’t perform log rotation, Ubuntu and CentOS rotate catalina.out on a weekly basis using logrotate.

The following example shows the output of an ArithmeticException logged by Tomcat. We log the exception using Tomcat’s global logging.properties file.

The servlet:

Navigating to the servlet’s page generates the following log entry:

The output is similar to what you would expect from a standard Java application. The first few lines show the date, Logger name, method name, and log message. The rest of the entry shows the complete stack trace, including messages from Tomcat’s other components.

Using Different Frameworks with Tomcat

Using a different logging framework with Tomcat means removing the hard-coded dependency on java.util.logging. This involves replacing JULI with an implementation that can detect and configure alternative frameworks. Apache provides a replacement with the JULI adapters and JULI log4j packages. The JULI log4j package replaces the tomcat-juli.jar file located at $CATALINA_HOME/bin/tomcat-juli.jar, while the JULI adapters package is added to $CATALINA_HOME/lib/. When using multiple instances of Tomcat, add the files to the $CATALINA_BASE directory.

Note that this process is necessary only if you’re logging from Tomcat itself. Your choice of a logging framework for your web applications is independent.

Log4j

Tomcat’s Log4j support is limited to version 1.2.x. Download and place the log4j.jar file in the $CATALINA_HOME/lib/ directory and add your log4j.properties file to $CATALINA_HOME/conf/logging.properties. Remove $CATALINA_HOME/conf/logging.properties to prevent java.util.logging from generating empty log files.

For more details, see Apache’s Tomcat documentation.

Logback and SLF4J

The tomcat-slf4j-Logback project bundles the Tomcat server, SLF4J, and Logback into a single unified package. It allows Tomcat to use SLF4J and Logback for its internal logging, while still allowing web applications to use their own SLF4J implementations. In addition to replacing tomcat-juli.jar, the installation requires you to create Logback-specific configuration files and update the server.xml file. You can find more information at the project’s website.

Additional Resources

Log Management Solutions

Guides and Tutorials

Tomcat

Written & Contributed by

Andre

Tony

This guide will help software developers and system administrators become experts at using logs to better run their systems. This is a vendor-neutral, community effort featuring examples from a variety of solutions

Meet Our Contributors Become a contributor