Support Logging setup Journald system and Docker logs

Journald system and Docker logs

This setup is useful for those who want to bypass syslog altogether when sending their journald daemon logs or docker container logs using journald docker logging driver. The journald to syslog to loggly approach is not optimal as it loses all the good metadata that journald contains. For example, when docker is forwarding its logs to journald, the CONTAINER_ID and CONTAINER_NAME are captured, which is not the case with the syslog approach.

Note: This setup is supported by systemd so please ensure you are on the latest Linux distribution that supports systemd and systemctl commands to start/stop/restart the system services.

Logging Setup

1. Configuring journald

We are going to use a systemd service which will be responsible for collecting the journald daemon logs and forwarding them to Loggly without excluding all the attached metadata. Please create a service file journald-forwarder.service under your /etc/systemd/system directory by running the command below:

sudo vi journald-forwarder.service

Paste the content  below content in the above created file:

[Unit]
Description=Forward journald logs to Loggly
After=docker.service

[Service]
ExecStartPre=-/bin/mkdir -pv /opt/loggly/journald-forwarder
ExecStartPre=-/usr/bin/curl -L -o /opt/loggly/journald-forwarder/journald-forwarder https://github.com/uswitch/journald-forwarder/releases/download/v0.4/journald-forwarder
ExecStartPre=-/bin/chmod +x /opt/loggly/journald-forwarder/journald-forwarder
ExecStart=/opt/loggly/journald-forwarder/journald-forwarder -token [token] -tag journald-loggly

Replace:

2. Reload units and start Journald

Since the journald-forwarder.service changed on disk, you may have to run the command below to reload units:

sudo systemctl daemon-reload

Now, copy and paste the following command in terminal to restart journald-forwarder.service:

sudo systemctl start journald-forwarder.service

You can check the status of the service by running the command below and it must be active and running:

sudo systemctl status journald-forwarder.service

Now your service is started and set up to monitor the journald daemon. It will forward all the logs that are coming to journald. To verify sending system logs run the command below.

logger Hello World

3. Verify Events

Search Loggly over the recent past to find logs with the journald-loggly tag . It may take a few minutes to index them. If you don’t see them, check the troubleshooting section below.

Collecting Docker Containers Logs

You can forward your docker container logs to Loggly by using the journald docker logging driver. The journald logging driver forwards docker container logs to systemd which then forwards them to Loggly using the journald-forwarder service.

Forwarding Nginx docker container logs

Since journald captures important metadata, we are going to demonstrate how to start a docker container using the jouranld logging driver and generate nginx server events to send them to Loggly.

At this point, we assume that you have your journald-forwarder service running already. Please run the command below to start your Nginx container which will be running on port 8080.

docker run --log-driver=journald -d -p 8080:80 nginx

You can see that we used the journald driver and it is exposed on port 8080. You can verify the running container by using the below command:

docker ps -a

Now generate some server logs by hitting the localhost URL below:

curl http://localhost:8080/testing

You can see the combined output of the above commands in the snippet below:

root@ip-172-31-48-18:/home/ubuntu# docker run --log-driver=journald -d -p 8080:80 nginx
b9b7f5b335d485b512e11653cb5bc89c5de30087ac4f4510fed088bbd9b9d1c5
root@ip-172-31-48-18:/home/ubuntu# docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                  NAMES
b9b7f5b335d4        nginx               "nginx -g 'daemon of…"   5 seconds ago       Up 5 seconds        0.0.0.0:8080->80/tcp   practical_kare
root@ip-172-31-48-18:/home/ubuntu# curl http://localhost:8080/testing

Verify Events

Search Loggly over the recent past for logs with the journald-loggly tag . It may take a few minutes to index them. If you don’t see them, check the troubleshooting section below.

In the event shown above, you can clearly see all the attached metadata related to the docker container like CONTAINER_NAME and CONTAINER_ID as well as the machine.

Advanced Options

  • You can confirm that your docker container is using the journald logging driver by running the command below:
    docker inspect -f '{{.HostConfig.LogConfig.Type}}' container_name

    Replace:

    • container_name: enter your docker container name.

    The output of the above command will be similar to the follwoing:

    root@ip-172-31-48-18:/home/ubuntu# docker ps -a
    CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                  NAMES
    b9b7f5b335d4        nginx               "nginx -g 'daemon of…"   About an hour ago   Up About an hour    0.0.0.0:8080->80/tcp   practical_kare
    root@ip-172-31-48-18:/home/ubuntu# docker inspect -f '{{.HostConfig.LogConfig.Type}}' practical_kare
    journald
    root@ip-172-31-48-18:/home/ubuntu#
  • Since we are using the journald logging driver, which supports the journalctl command to see the container’s logs locally and for many other debugging purposes, we can also check our nginx container’s logs locally by running the command below:
    journalctl CONTAINER_NAME=container_name

    Replace:

    • container_name: enter your docker container name.

    Optional: You can also put your container’s ID in place of the container name.

    See the example below:

    root@ip-172-31-48-18:/home/ubuntu# docker ps -a
    CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                  NAMES
    b9b7f5b335d4        nginx               "nginx -g 'daemon of…"   34 minutes ago      Up 34 minutes       0.0.0.0:8080->80/tcp   practical_kare
    root@ip-172-31-48-18:/home/ubuntu# journalctl CONTAINER_NAME=practical_kare
    -- Logs begin at Fri 2018-03-23 12:32:24 UTC, end at Thu 2018-03-29 12:25:16 UTC. --
    Mar 29 11:53:31 ip-172-31-48-18 b9b7f5b335d4[4116]: 2018/03/29 11:53:31 [error] 5#5: *1 open() "/usr/share/nginx/html/testing" failed (2: No such file or directory), cl
    Mar 29 11:53:31 ip-172-31-48-18 b9b7f5b335d4[4116]: 172.17.0.1 - - [29/Mar/2018:11:53:31 +0000] "GET /testing HTTP/1.1" 404 170 "-" "curl/7.47.0" "-"
    Mar 29 12:03:40 ip-172-31-48-18 b9b7f5b335d4[4116]: 2018/03/29 12:03:40 [error] 5#5: *2 open() "/usr/share/nginx/html/testing" failed (2: No such file or directory), cl
    Mar 29 12:03:40 ip-172-31-48-18 b9b7f5b335d4[4116]: 172.17.0.1 - - [29/Mar/2018:12:03:40 +0000] "GET /testing HTTP/1.1" 404 170 "-" "curl/7.47.0" "-"
    root@ip-172-31-48-18:/home/ubuntu#
    

Troubleshooting Journald

  • Make sure your journald-forwarder service is active and running.
  • Make sure you have replaced your customer token in the journald-forwarder.service file correctly.
  • Search or post your own question in the community forum.
Thanks for the feedback! We'll use it to improve our support documentation.