Troubleshooting Rsyslog

Use these tips to troubleshoot problems with Rsyslog. You can use our automated test, check the configuration, send sample data, and check transmission. Additionally, you can read the Rsyslog manual, try their support forum (they offer professional Rsyslog support), or check out our Rsyslog manual configuration docs.

Wait a Few Minutes

Wait a few minutes after sending an event to give it time to index and appear in the search results. It normally happens within seconds, but sometimes it can take longer.

Check Loggly Status

If Loggly isn’t seeing data check our status page to make sure we are indexing data and search is running. You should see green dots and “All Systems Operational”.

Automated Verification

Our configure-syslog script can send a test event to Loggly, and then verify if it’s received using the Loggly search API. You can overwrite your existing loggly configuration to make sure there are no errors and verify it again. It may take a few minutes to run.

curl -O https://www.loggly.com/install/configure-linux.sh
sudo bash configure-linux.sh -a SUBDOMAIN -u USERNAME

Replace:

  • SUBDOMAIN: your account subdomain that you created when you signed up for Loggly
  • USERNAME: your Loggly username

Check Rsyslog Configuration

Make sure you restarted rsyslog so your changes take effect

sudo service rsyslog restart

Make sure rsyslog is running. If this command returns nothing than it’s not running.

ps -A | grep rsyslog

Check the rsyslog configuration. If there are no errors listed, then it’s ok.

rsyslogd -N1

Make sure you have Rsyslog version 5.8 or higher

rsyslogd -version

Check the Linux system log for rsyslog errors. You should see an event that it started and no errors. Some logs may also be in /var/log/syslog.

sudo cat /var/log/messages | grep rsyslog

Make sure Loggly is configured in your rsyslog configuration. There should be an endpoint for logs-01.loggly.com either in your main rsyslog.conf file or an include to the 22-loggly.conf file.

sudo vim /etc/rsyslog.d/22-loggly.conf

Check the permissions of rsyslog and the file you want to monitor to be sure it can read that file. You may need to use alter the privilege in the rsyslog.conf file:

$PrivDropToUser adm

Send Sample Data

Verify rsyslog is sending data to Loggly by making a test event. Then search for that event in Loggly by searching for “TroubleshootingTest” in the last hour.

logger TroubleshootingTest

Check the Linux system log to see if Rsyslog recorded the test event

sudo cat /var/log/messages | grep TroubleshootingTest

Rsyslog

If you are sending repeated test messages, you should turn off repeated message reduction in the rsyslog configuration.

$RepeatedMsgReduction off

If you are filtering events out with a lower priority, you should send test events with a high enough priority.

logger -p local0.error "TroubleshootingTest"

Check Data Transmission

Use netstat to verify Rsyslog has an established connection to Loggly. Specifically, check that Loggly can make a connection through your firewall on the proper port. It’s 514 for syslog, 6514 for TLS syslog, 80 for HTTP, and 643 for HTTPS.

sudo netstat -taupn | grep syslog

Rsyslog Configuration
Use telnet to verify we can make an outbound connection to Loggly. For syslog we use port 514. If you can’t connect it might be a network or firewall issue.

telnet logs-01.loggly.com 514

Rsyslog Tutorial
Use tcpdump to verify data is being sent to Loggly. If you send your events in cleartext while tcpdump is running, you should be able to see them in the left hand column.

sudo tcpdump -A dst logs-01.loggly.com

Rsyslog Logging
If your application logs syslog to rsyslog, you can also test to see if messages making it to rsyslog over UDP to localhost.

sudo tcpdump -i lo -A udp and port 514

Check Log Rotation

Some older version of rsyslog may have trouble resuming after a log is rotated. If you have log rotation setup, follow these instructions to force rsyslog to pick up the new file.

Read More

  • Rsyslog-users – Mailing list for rsyslog describing many common support issues

Still Not Working?

Please search our community forum for more Rsyslog configuration answers or post your own question.

Rsyslog

Thanks for the feedback! We'll use it to improve our support documentation.


Top