LoggingThe Ultimate Guide

your open-source resource for understanding, analyzing, and troubleshooting system logs

curated byloggly

10

Linux Logging with Systemd

Systemd is the new system and service manager for Linux. It has become the de facto system management daemon in various Linux distributions in recent years. Systemd was first introduced in Fedora. Other distributions like Arch Linux, openSUSE or CoreOS have already made it part of their operating systems. Red Hat Enterprise Linux (RHEL) and its downstream distros like CentOS started to use systemd natively from version 7. Another major distribution, Ubuntu—which had introduced another service management daemon called Upstart—now ships with systemd from version 15.

The reason for this wide-scale adoption is the versatility of systemd. It manages not only daemons and processes in a Linux system but also various resources like devices, sockets or mount points. When the system boots, it does not load services sequentially like System V, which saves significant time at startup. Services are loaded in parallel, and a service waits till other required resources for it are also activated.

Systemd is backward compatible with its predecessors like System V init and Upstart. That means any service still using older System V init scripts for starting will work under systemd and you can use systemd commands like systemctl to start, stop and check the service’s status. Another advantage of systemd is its ease of configuration. Systemd is controlled by unit files that are declarative in nature and easy to understand. This contrasts with System V where the application’s developer had to create complicated shell scripts for starting, stopping or reloading the service.

As we will see later, systemd has a sophisticated logging service that can be used instead of the traditional syslog service. It can also be used to complement syslog.

Units and Targets

At the heart of systemd are unit files. A unit file is a plain text file that lives under /lib/systemd/system directory and has a type associated with it. A unit file basically describes a resource and tells systemd how to activate that resource. The naming standard for a unit file is <resource_name>.<unit_type>. The different types of units include service, path, mount point, automount, swap, target, timer, device and socket. So, we have unit files like cron.service, tmp.mount, syslog.socket or graphical.target. For each service unit that’s enabled, a symbolic link to the unit file is placed under the /etc/systemd/system/<target>.wants/ directory.

A target unit is a special kind of unit file because it does not represent a single resource; rather, it groups other units to bring the system to a particular state. Target units in systemd loosely resemble run levels in System V in the sense that each target unit represents a particular system state. For example, the graphical.target unit represents a system that has booted in multi-user, graphical mode, similar to System V’s runlevel5. Multi-user.target, on the other hand, is similar to runlevel3 (multi-user, text mode with networking enabled). However, targets are also different from run levels because in System V, a Linux box can exist in only one run level at any time. In systemd, target units are inclusive. A target unit can group other target units when it’s coming up — so it’s possible for a system to remain in more than one target. Going back to the graphical.target example, when the target comes up, it also activates multi-user.target.
For a good introduction on systemd, you can refer to this article from DigitalOcean, or this e-book.

Systemd Journal Basics

The journal is a component of systemd. It’s a centralized location for all messages logged by different components in a systemd-enabled Linux system. This includes kernel and boot messages, messages coming from syslog or different services.

In traditional Linux, the boot-up phase, different subsystems of the OS or application daemons would log all their message in different text files throughout the system. Each system would log its messages with varying level of details. When troubleshooting, an administrator would often have to go through messages from multiple files within different time frames and correlate the entries. Journaling takes away this difficulty by recording both OS and application level messages in one place.

The journal is controlled by the systemd-journald daemon. It collects information from different sources and loads the messages in the journal.

The systemd journal is not a large text file. It’s a binary file maintained by the daemon. So it can’t be opened with a text editor. As we will see later, the location and size of this binary file is controlled by the daemon’s configuration file. It does not have to be persistent either: using configuration parameters, an administrator can turn off journaling altogether or keep it in memory so it’s volatile in nature. With in-memory journaling, systemd creates its journal files under the /run/log/journal directory. The directory is created if it does not exist. With persistent storage, the journal is created under /var/log/journal directory; again, the directory is created by systemd if needed. If this directory is deleted for some reason, systemd-journald will not re-create it automatically; rather, it will write the logs under /run/log/journal in a non-persistent way. It will re-create the directory when the daemon is restarted.

Here is an example of the systemd journal:

ls -l /var/log/journal drwxr-sr-x 2 root systemd-journal 4096 Jun 25 00:06 fd8cf26e06e411e4a9d004010897bd01 ls -l /var/log/journal/fd8cf26e06e411e4a9d004010897bd01/ -rw-r—– 1 root root 109051904 Jun 27 23:00 system.journal

With systemd journal, there is no option or reason for a traditional syslog utility like logrotate. Systemd-journald can be configured to grow its files up to a percentage of the size of the volume it’s hosted in. The daemon would then automatically delete old journal entries to keep the size below that threshold. Again, as we will see later, there are multiple options for controlling journal size.

Journald Configuration

The main configuration file for systemd-journald is /etc/systemd/journald.conf. However, other packages can create their configuration files which can be under any of these directories with a .conf extension:

  • /etc/systemd/journald.conf.d/*.conf
  • /run/systemd/journald.conf.d/*.conf
  • /usr/lib/systemd/journald.conf.d/*.conf

The main configuration file is read before any of the custom *.conf files. If there are custom configs present, they override the main configuration parameters.

A look into the default configuration file shows the following entries. As you can see, all the parameters are commented out, meaning the values are already known to systemd as default values. If any of the values need to be changed, they have to be uncommented and the systemd-journald.service restarted.

A brief description of some of the configuration parameters are shown below. The parameters relate to the:

  1. Persistence of Journal event messages
  2. Managing disk space
  3. Forwarding to other syslogs and other output media
Parameter Purpose and Possible Values
Storage There are four possible values for it:
“none”: This effectively turns off journaling. Any log message received will be dropped. However, any redirection to console or syslog or kernel log buffer would still be in effect.
“volatile”: Journal data is saved in memory and temporarily available under the /run/log/journal directory. The directory will be created if it does not exist.
“persistent”: Journal data is saved persistently on disk under the /var/log/journal directory. The directory will be created if it does not exist. If the disk volume is not accessible or writable, the files will be created under /run/log/journal.
“auto”: The storage mode is like persistent — data will be written to disk; however, if the /var/log/journal directory does not exist, it will be created under /run/log/journal.
Compress If this parameter is enabled, data stored in the journal that is larger than a threshold will be compressed before being written to disk. The option is turned on by default.
SystemKeepFree This is is one of the several parameters that control how large the journal can grow up to. This parameter applies if systemd is saving journals under the /var/log/journal directory. It specifies how much disk space the systemd-journald daemon will leave for other applications in the file system where the journal is hosted. The default is 15%.
RuntimeKeepFree Same as SystemKeepFree, except this applies when the journaling storage option is set to “volatile”, meaning journal files are created under /run/log/journal.
SystemMaxuse This parameter controls the maximum disk space the journal can consume when it’s persistent. This defaults to 10% of the disk space.
RuntimeMaxUse Same as SystemMaxUse, applicable for volatile journal storage (when files are saved under /run/log/journal). Again, the default is 10%.
SystemMaxFileSize This specifies the maximum size of a journal file for persistent storage. This defaults to one-eighth of the size of the SystemMaxUse parameter.
RuntimeMaxFileSize Same as SystemMaxFileSize: applies to files under /run/log/journal.
MaxRetentionSec The maximum time to store entries in the journal. Journal files containing records which are older than this period will be deleted automatically. However, the value does not have to be specified in seconds only. It can be suffixed with “year”, “month”, “week”, “day”, “h” or “m”. The default is 0.
This parameter does not need to be set from the default value of 0 because the SystemMaxUse parameter will make sure the journal does not grow and fill up the entire disk space.
MaxFileSec This is same as the MaxRetentionSec parameter, except this applies to individual journal files. This parameter has a default value of one month and controls the maximum time to keep journal entries in a single journal file. Again, this can be turned off with a value of 0 because the parameter SystemMaxFileSize can control an individual file’s maximum size.
ForwardToSyslog This parameter specifies if log messages that are received by the systemd-journald daemon will also be forwarded to a syslog daemon. The default is yes, but if no process is reading off from the socket, nothing happens.
ForwardToWall If log messages received by systemd-journald will be also be sent as wall messages to all logged-in users. Default is yes.
ForwardToKMsg If log messages received by systemd-journald will also be forwarded to the kernel log buffer. Default is no.
ForwardToConsole If log messages received by systemd-journald will also be forwarded to the system console.
If this parameter is enabled, another parameter TTYPath determines the console TTY to send messages to. The default value of that parameter is /dev/console.
MaxLevelStore This parameter can take any of the following values:

  • 0 or “emerg”
  • 1 or “alert”
  • 2 or “crit”
  • 3 or “err”
  • 4 or “warning”
  • 5 or “notice”
  • 6 or “info”
  • 7 or “debug”

All messages equal or below the level specified will be stored on disk. The default value is “debug” which means all log messages from “emerg” to “debug”.

MaxLevelSyslog Controls the maximum level of log message forwarded to Syslog. Again the default value is “debug”.
MaxLevelKMsg Controls the maximum level of log message forwarded to the kernel log buffer. The default value is “notice”.
MaxLevelConsole Controls the maximum level of log message forwarded to the system console with a default value of “info”.
MaxLevelWall Controls the maximum level of log message forwarded to the walls of logged-in users. Defaults to “emerg” — meaning if enabled, users will be immediately notified of emergency events.
  • James triplett

    how to read the logs when the system is down? My system won’t boot. In the old days, I could mount the disk somewhere else and just look at the log text. How do you do this using journalctl?

    • Randy Legault

      Yes — I would like a way to mirror the journal messages to an external hard-drive for such cases of a frozen system – as happened this a.m.

  • Sadequl Hussain

    Hi James,

    There are some ways you can still try to recover the system journals. I will try to provide the solution below from a RHEL-based distro perspective, but the principles should apply to other distros as well.

    Use Case 1: journal entries copied to syslog
    ===================================

    By default sysdemd also streams journal entries to syslog. If you know this was the case for your system, you can easily detach the file system hosting the syslog file (/var/log/message), attach it to an another node and check the file. This is the classic method you mentioned.

    Now if you don’t know if systemd-journald was configured to send journal entries to syslog (which is the common case), you can check the /etc/systemd/journald.conf file or

    any of the these locations for custom application journal configurations:

    /etc/systemd/journald.conf.d/*.conf
    /run/systemd/journald.conf.d/*.conf
    /usr/lib/systemd/journal.conf.d/*.conf

    The parameter you would want to check from the config file would be “ForwardToSyslog”. If it is yes (or null), then you are good to get the syslog file.

    Obviously you can check the config file only if you detach and reattach the root volume (or wherever /etc was hosted in) from your downed instance to a new machine.

    Use Case 2: journal entries are not copied to syslog
    =======================================

    Now suppose your journal entries were not being redirected to syslog. You still can try to recover using the steps below. However, this will work only if journaling was not configured to be volatile: the log entries were persisted in a journal file. With persistent journaling, journal is created under the /var/log/journal directory.

    So the steps would be:

    1. Detach and attach the volume containing the /var/log directory from your old node to your new node

    2. Check if the journal file exists under /var/log/journal directory

    3. Use the journalctl command with the –D switch or –directory directive.

    From journalctl reference: “-D DIR, –directory=DIR takes a directory path as argument. If specified, journalctl will operate on the specified journal directory DIR instead of the default runtime and system journal paths.”

    Bear in mind that older journal entries are deleted to keep the file from taking up all the disk space and so if you want to go back to a date that is too far behind, you may not find it in the logs.

    Hope this helps. Let me know if this answers your question.

    • https://nick3499.github.io/ Nick

      A number of times, I messed around with my Ubuntu OS enough to prevent myself from logging onto the system. Whenever having been locked out of it. Your post gives me the impression that a corrupted journal may have been a typical issue. In other words, I may have corrupted my Ubuntu journal which locked me out of my system. So journal recovery seems to be the first thing I should try. To find my breadcrumb trail.

  • Mounesh

    Hi Sadequl,
    I have a question:

    I have set RuntimeMaxUse=32M. With this value, I am expecting 7 archive
    files and one active journal file. But I am getting 8 archive files each
    of 4M and one active journal file (4M). So in total it is taking 36M,
    instead of RuntimeMaxUse=32M.

    Why is it so ?

  • Liviu Stancu

    Hello there, I have a problem at the system start-up with the systemd-logind service which fails the first time it is executed by systemd, but on the second retry it starts ok. According to https://lists.freedesktop.org/archives/systemd-devel/2013-March/010004.html , the debug level refers to logging for systemd itself. I have tried
    to copy /usr/lib/systemd/system/systemd-logind.service to
    /etc/systemd/system/systemd-logind.service and then edit it there as suggested but still I can’t get any additional systemd-logind logs. Any idea on how to do this?

  • Eric S

    so:
    sudo tail -f /var/log/messages
    is now worthless… is there any equivalent?!?!?!

    • rtfazeberdee

      Have a look at journalctl and its options to read the journal records – its very powerful

    • Andy Airey

      Yes, journalctl -f

  • Zoran Destroyer

    Arcticle is excellent.
    Question is: How to forward log messages to remote syslogd?

Written & Contributed by

Sadequl

Systemd is the new system and service manager for #Linux: http://bit.ly/1ODBaR1 @loggly

Systemd has a sophisticated #logging service that can be used instead of the traditional syslog service: http://bit.ly/1ODBaR1 @loggly

This guide will help software developers and system administrators become experts at using logs to better run their systems. This is a vendor-neutral, community effort featuring examples from a variety of solutions

Meet Our Contributors Become a contributor