Kubernetes Logging Through Logspout
You can send Kubernetes pod logs to Loggly by using the popular open source tool called Logspout. Logspout pulls logs from Docker’s standard json log files. By using a DaemonSet, Logspout will be deployed across all nodes in your cluster, allowing logs from all nodes to be aggregated in Loggly. It does not allow multiline logs so each log line will be treated as a separate logging event. This setup has been tested with Loggly v3.2.4 and Kubernetes 1.7 and up.
- Loggly account
- Kubernetes cluster
Kubernetes Logspout Setup
1. Create a secret for your Loggly customer token
From the Loggly product, click on “Source Setup” in the main menu, and then “Customer Tokens” in the sub-menu to get to the customer tokens page. Use this token to create a Kubernetes secret containing the syslog structured data and endpoint for Logspout to use.
kubectl create secret generic logspout-config --from-literal=syslog-structured-data="$TOKEN@41058 tag=\"kubernetes\""
- $TOKEN: enter your customer token before creating the secret.
2. Create the Logspout DaemonSet
A Kubernetes DaemonSet is a control structure for starting an instance of a pod on all nodes in a Kubernetes cluster. As your cluster scales, you won’t need to worry about starting more Logspout instances, since Kubernetes will take over starting and restarting pods as needed.
Run the command below to create a DaemonSet and start the Logspout pod on all of your Kubernetes nodes. The linked gist contains the DaemonSet manifest configured to use the secret we created in the first step, as well as annotations so that Loggly can parse the application and pod names as logs are received. You may run it as is or download it and customize it for your needs.
kubectl apply -f https://gist.githubusercontent.com/gerred/3cac803b9f8f581b22f562b509d897cf/raw/41cdccdc9364c3e82f9f90d925cc9b67417bf762/logspout-ds.yaml
3. Verify Events
Go to the Search tab and search for events in the past 10 minutes. It may take a few minutes for incoming events to appear.
Aggregation of logs for multiple pods
Logspout collects logs of every Docker container running on the system and delivers them to Loggly. Logs for an individual pod are parsed as syslog.appName, while logs for all pods in a given application are parsed as syslog.host. When aggregating logs across multiple pods, key searches off of the syslog.host field. This will ensure that all logs for an application are captured.
Example: Centralizing nginx server logs with Logspout
Step 1. Check pod status
You should already have created the Logspout DaemonSet and can see the running pod status by running the command below:
[ec2-user@ip-10-0-0-244 ~]$ kubectl get pods -l name=logspout NAME READY STATUS RESTARTS AGE logspout-pwbw2 1/1 Running 0 17m
Step 2. Run Nginx container
Run the nginx container and expose it as a service using the command below.
kubectl run nginx --image nginx --port='80' --expose
Step 3. Start the kubectl proxy
Start the kubectl proxy and access the service proxy endpoints a few times to generate a few logs:
kubectl proxy & curl https://localhost:8001/api/v1/namespaces/default/services/nginx/proxy/
Search Loggly for events with the syslog.nginx host over the last 10 minutes. It may take a few minutes for events to be indexed. If no events show up, see the troubleshooting section below.
Expanding each individual log will show the details for a log event.
In the example above, Loggly has detected that these are nginx logs and parsed them to provide better information. The logtype will be set to both nginx and syslog representing this parsing, while the tag will be set to `kubernetes`. This tag can be changed by changing the tag in the syslog-structured-data part of the secret you created in the first setup step.
- Loggly Libraries Catalog: Bypass Docker and log directly from an application directly using our HTTP/S libraries.
Kubernetes Logging Troubleshooting
If no data appears in the verification steps, check for these common problems:
Check Kubernetes Pods:
- Wait a few minutes in case indexing needs to catch up
- Verify the pod is running by running `kubectl get pods -l name=logspout`
- Make sure it can access the Loggly host by getting the logspout logs: `kubectl logs $POD_NAME`. Remember to replace $POD_NAME with the pod from the last troubleshooting step
- You can only relay logs with Logspout on Kubernetes clusters using the Docker with the json-file/journald drivers. Currently, containerd is untested and unsupported.
Still Not Working?
- Search or post your own Kubernetes logging question in the community forum.