Linux Logging with Systemd
Systemd is the new system and service manager for Linux. It has become the de facto system management daemon in various Linux distributions in recent years. Systemd was first introduced in Fedora. Other distributions like Arch Linux, openSUSE or CoreOS have already made it part of their operating systems. Red Hat Enterprise Linux (RHEL) and its downstream distros like CentOS started to use systemd natively from version 7. Another major distribution, Ubuntu—which had introduced another service management daemon called Upstart—now ships with systemd from version 15.
The reason for this wide-scale adoption is the versatility of systemd. It manages not only daemons and processes in a Linux system but also various resources like devices, sockets or mount points. When the system boots, it does not load services sequentially like System V, which saves significant time at startup. Services are loaded in parallel, and a service waits till other required resources for it are also activated.
Systemd is backward compatible with its predecessors like System V init and Upstart. That means any service still using older System V init scripts for starting will work under systemd and you can use systemd commands like systemctl to start, stop and check the service’s status. Another advantage of systemd is its ease of configuration. Systemd is controlled by unit files that are declarative in nature and easy to understand. This contrasts with System V where the application’s developer had to create complicated shell scripts for starting, stopping or reloading the service.
As we will see later, systemd has a sophisticated logging service that can be used instead of the traditional syslog service. It can also be used to complement syslog.
Units and Targets
At the heart of systemd are unit files. A unit file is a plain text file that lives under /lib/systemd/system directory and has a type associated with it. A unit file basically describes a resource and tells systemd how to activate that resource. The naming standard for a unit file is <resource_name>.<unit_type>. The different types of units include service, path, mount point, automount, swap, target, timer, device and socket. So, we have unit files like cron.service, tmp.mount, syslog.socket or graphical.target. For each service unit that’s enabled, a symbolic link to the unit file is placed under the /etc/systemd/system/<target>.wants/ directory.
A target unit is a special kind of unit file because it does not represent a single resource; rather, it groups other units to bring the system to a particular state. Target units in systemd loosely resemble run levels in System V in the sense that each target unit represents a particular system state. For example, the graphical.target unit represents a system that has booted in multi-user, graphical mode, similar to System V’s runlevel5. Multi-user.target, on the other hand, is similar to runlevel3 (multi-user, text mode with networking enabled). However, targets are also different from run levels because in System V, a Linux box can exist in only one run level at any time. In systemd, target units are inclusive. A target unit can group other target units when it’s coming up — so it’s possible for a system to remain in more than one target. Going back to the graphical.target example, when the target comes up, it also activates multi-user.target.
For a good introduction on systemd, you can refer to this article from DigitalOcean, or this e-book.
Systemd Journal Basics
The journal is a component of systemd. It’s a centralized location for all messages logged by different components in a systemd-enabled Linux system. This includes kernel and boot messages, messages coming from syslog or different services.
In traditional Linux, the boot-up phase, different subsystems of the OS or application daemons would log all their message in different text files throughout the system. Each system would log its messages with varying level of details. When troubleshooting, an administrator would often have to go through messages from multiple files within different time frames and correlate the entries. Journaling takes away this difficulty by recording both OS and application level messages in one place.
The journal is controlled by the systemd-journald daemon. It collects information from different sources and loads the messages in the journal.
The systemd journal is not a large text file. It’s a binary file maintained by the daemon. So it can’t be opened with a text editor. As we will see later, the location and size of this binary file is controlled by the daemon’s configuration file. It does not have to be persistent either: using configuration parameters, an administrator can turn off journaling altogether or keep it in memory so it’s volatile in nature. With in-memory journaling, systemd creates its journal files under the /run/log/journal directory. The directory is created if it does not exist. With persistent storage, the journal is created under /var/log/journal directory; again, the directory is created by systemd if needed. If this directory is deleted for some reason, systemd-journald will not re-create it automatically; rather, it will write the logs under /run/log/journal in a non-persistent way. It will re-create the directory when the daemon is restarted.
Here is an example of the systemd journal:
ls -l /var/log/journal drwxr-sr-x 2 root systemd-journal 4096 Jun 25 00:06 fd8cf26e06e411e4a9d004010897bd01 ls -l /var/log/journal/fd8cf26e06e411e4a9d004010897bd01/ -rw-r—– 1 root root 109051904 Jun 27 23:00 system.journal
With systemd journal, there is no option or reason for a traditional syslog utility like logrotate. Systemd-journald can be configured to grow its files up to a percentage of the size of the volume it’s hosted in. The daemon would then automatically delete old journal entries to keep the size below that threshold. Again, as we will see later, there are multiple options for controlling journal size.
The main configuration file for systemd-journald is /etc/systemd/journald.conf. However, other packages can create their configuration files which can be under any of these directories with a .conf extension:
The main configuration file is read before any of the custom *.conf files. If there are custom configs present, they override the main configuration parameters.
A look into the default configuration file shows the following entries. As you can see, all the parameters are commented out, meaning the values are already known to systemd as default values. If any of the values need to be changed, they have to be uncommented and the systemd-journald.service restarted.
# This file is part of systemd.## systemd is free software; you can redistribute
# it and/or modify it under the terms of the GNU Lesser General Public License as
# published by the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
# See journald.conf(5) for details.
A brief description of some of the configuration parameters are shown below. The parameters relate to the:
- Persistence of Journal event messages
- Managing disk space
- Forwarding to other syslogs and other output media
|Parameter||Purpose and Possible Values|
|Storage||There are four possible values for it:
“none”: This effectively turns off journaling. Any log message received will be dropped. However, any redirection to console or syslog or kernel log buffer would still be in effect.
“volatile”: Journal data is saved in memory and temporarily available under the /run/log/journal directory. The directory will be created if it does not exist.
“persistent”: Journal data is saved persistently on disk under the /var/log/journal directory. The directory will be created if it does not exist. If the disk volume is not accessible or writable, the files will be created under /run/log/journal.
“auto”: The storage mode is like persistent — data will be written to disk; however, if the /var/log/journal directory does not exist, it will be created under /run/log/journal.
|Compress||If this parameter is enabled, data stored in the journal that is larger than a threshold will be compressed before being written to disk. The option is turned on by default.|
|SystemKeepFree||This is is one of the several parameters that control how large the journal can grow up to. This parameter applies if systemd is saving journals under the /var/log/journal directory. It specifies how much disk space the systemd-journald daemon will leave for other applications in the file system where the journal is hosted. The default is 15%.|
|RuntimeKeepFree||Same as SystemKeepFree, except this applies when the journaling storage option is set to “volatile”, meaning journal files are created under /run/log/journal.|
|SystemMaxuse||This parameter controls the maximum disk space the journal can consume when it’s persistent. This defaults to 10% of the disk space.|
|RuntimeMaxUse||Same as SystemMaxUse, applicable for volatile journal storage (when files are saved under /run/log/journal). Again, the default is 10%.|
|SystemMaxFileSize||This specifies the maximum size of a journal file for persistent storage. This defaults to one-eighth of the size of the SystemMaxUse parameter.|
|RuntimeMaxFileSize||Same as SystemMaxFileSize: applies to files under /run/log/journal.|
|MaxRetentionSec||The maximum time to store entries in the journal. Journal files containing records which are older than this period will be deleted automatically. However, the value does not have to be specified in seconds only. It can be suffixed with “year”, “month”, “week”, “day”, “h” or “m”. The default is 0.
This parameter does not need to be set from the default value of 0 because the SystemMaxUse parameter will make sure the journal does not grow and fill up the entire disk space.
|MaxFileSec||This is same as the MaxRetentionSec parameter, except this applies to individual journal files. This parameter has a default value of one month and controls the maximum time to keep journal entries in a single journal file. Again, this can be turned off with a value of 0 because the parameter SystemMaxFileSize can control an individual file’s maximum size.|
|ForwardToSyslog||This parameter specifies if log messages that are received by the systemd-journald daemon will also be forwarded to a syslog daemon. The default is yes, but if no process is reading off from the socket, nothing happens.|
|ForwardToWall||If log messages received by systemd-journald will be also be sent as wall messages to all logged-in users. Default is yes.|
|ForwardToKMsg||If log messages received by systemd-journald will also be forwarded to the kernel log buffer. Default is no.|
|ForwardToConsole||If log messages received by systemd-journald will also be forwarded to the system console.
If this parameter is enabled, another parameter TTYPath determines the console TTY to send messages to. The default value of that parameter is /dev/console.
|MaxLevelStore||This parameter can take any of the following values:
All messages equal or below the level specified will be stored on disk. The default value is “debug” which means all log messages from “emerg” to “debug”.
|MaxLevelSyslog||Controls the maximum level of log message forwarded to Syslog. Again the default value is “debug”.|
|MaxLevelKMsg||Controls the maximum level of log message forwarded to the kernel log buffer. The default value is “notice”.|
|MaxLevelConsole||Controls the maximum level of log message forwarded to the system console with a default value of “info”.|
|MaxLevelWall||Controls the maximum level of log message forwarded to the walls of logged-in users. Defaults to “emerg” — meaning if enabled, users will be immediately notified of emergency events.|