Support Logging setup

Docker logging through Logspout

Docker logging through Logspout

You can send Docker container logs to Loggly with the help of Logspout. This setup doesn’t support multiline logs, each line will be treated as a separate log event. This setup allows sending logs from multiple containers to a centralized guest container from which Logspout can then send to Loggly. All the logs from the guest container will be forwarded to Logspout automatically until the guest container is restarted or exited, in which case, you will have to repeat the procedure again for centralizing logging. This setup has been tested with Docker version 17.03.2-ce and Logspout version 3.2.6.

Docker Logspout Setup

Run the Logspout container

Run the command below to start the Logspout container and configure it to send logs to Loggly.

docker run --name logspout -d --volume=/var/run/docker.sock:/var/run/docker.sock -e SYSLOG_STRUCTURED_DATA="TOKEN@41058 tag=\"Logspout\"" gliderlabs/logspout syslog+tcp://logs-01.loggly.com:514

Replace:

  • TOKEN: enter your customer token from the Source Setup page in Loggly

Verify Events

Go to the Search tab in Loggly and search for events with the Logspout tag over the last 20 minutes. It may take a few minutes to index the events. If it doesn’t work, see the troubleshooting section below.

tag:Logspout
Docker Syslog Example

Note: The container name and ID of the container will be parsed as syslog.appname and syslog.host respectively.

Centralization of logs from multiple containers

If you want to send multiple container logs to Loggly, you don’t need to set up each container to send the logs. The already running Logspout container will take care of other container’s logs and will forward them with its own container logs to Loggly.

“The logspout container has the ability to gather logs from other containers that are started without the -t option and are configured with a logging driver that works with docker logs (journald and json-file).”

Example for centralizing nginx server logs with Logspout

Step 1. You should  already have started the Logspout container and you can see the running container status by running the command below:

root@ip-172-31-57-234:/home/ubuntu# docker ps -a
CONTAINER ID        IMAGE                 COMMAND                  CREATED             STATUS              PORTS               NAMES
4280a44b3a0a        gliderlabs/logspout   "/bin/logspout sys..."   3 minutes ago       Up 3 minutes        80/tcp              logspout
root@ip-172-31-57-234:/home/ubuntu#

Step 2. Simply Run the nginx container using the command below.

docker run --name nginx -d -p 8080:80 nginx

You can now see that both Logspout and nginx containers are running.

root@ip-172-31-57-234:/home/ubuntu# docker ps -a
CONTAINER ID        IMAGE                 COMMAND                  CREATED             STATUS              PORTS                  NAMES
9f6e5e63fb4c        nginx                 "nginx -g 'daemon ..."   1 second ago        Up 1 second         0.0.0.0:8080->80/tcp   nginx
4280a44b3a0a        gliderlabs/logspout   "/bin/logspout sys..."   4 minutes ago       Up 4 minutes        80/tcp                 logspout
root@ip-172-31-57-234:/home/ubuntu#

Step 3. Your nginx container has started and is open to listen at port 8080. Now you are all set to generate nginx server logs by hitting the localhost URL with the 8080 port and logspout will ship all the logs to Loggly.

Verify Events

Search Loggly for events with the Logspout tag over the last 20 minutes. It may take a few minutes to index the events. If it doesn’t work, see the troubleshooting section below.

tag:Logspout

You should see tag: Logspout which shows that nginx server logs are coming from the Logspout container. The logtype is nginx because the logs are nginx. You will also notice the container name and ID in syslog.appName and syslog.host, respectively. You can easily differentiate between different containers and their logs by looking at the container IDs.

Multiline Support

In order to enable multiline logging, you must first prefix your adapter with the multiline adapter as shown below:

docker run --name logspout -d --volume=/var/run/docker.sock:/var/run/docker.sock -e SYSLOG_STRUCTURED_DATA="TOKEN@41058 tag=\"Logspout\"" gliderlabs/logspout multiline+syslog+tcp://logs-01.loggly.com:514

Replace:

  • TOKEN: enter your customer token from the Source Setup page in Loggly

Adding multiline adapter will allow you to see your multiline events like error stacktraces into a single event. Please see the screenshot below  to get an idea of it.

Note: If you are not using the multiline adapter and logging a multiline event or error stacktrace then you will see each line of your stacktrace as a single event in Loggly.

Advanced Options

Docker Logging Troubleshooting

If you don’t see any data show up in the verification step, check for these common problems.

Check Docker Container:

  • Wait a few minutes in case indexing needs to catch up
  • Verify the container is running by running sudo docker ps -a
  • Make sure your machine has Rsyslog installed on it
  • Send test events from inside each of the containers and from the host to see which point in the chain is dropping logs
  • You can only relay the other container’s logs to logspout which supports “json-file” or “journald” driver because only “json-file” and “journald” driver supports the “docker logs” command. Read more on limitations-of-logging-drivers

Still Not Working?

  • Search or post your own Docker logs, Docker daemon, or other Docker question in the community forum.
Thanks for the feedback! We'll use it to improve our support documentation.