Gen1: Logging with rsyslog

[obsolete]Loggly Generation 2 rsyslog configuration is available in our Knowledge Base.

Table of Contents

Rsyslog is an open source alternative to the standard syslog that's commonly found on UNIX and Unix-like systems (*nix). It uses the basic syslog protocol and complies with RFC 5424, it extends on the old syslogd with rich filtering capabilities, flexible configuration options and adds important features such as using TCP (As well as TLS) which is much more reliable than UDP. We highly recommend using rsyslog over standard syslog for those reasons.

The last couple versions of Ubuntu have included rsyslog as the default Syslog daemon. Read about the reasoning behind this decision.

Configuring rsyslog to Forward to Loggly

rsyslog supports both UDP and TCP (as well as TLS) to send data. Before we dive into the awesomeness that is rsyslog, make sure that you've created a Syslog TCP, UDP, or Secure Syslog input and take note of the port number Loggly provides. (If you're interested in using JSON formatting for some of your logs make sure to check the "JSON" box to enable it when creating that input)

 

This will send all log entries that rsyslog manages to Loggly:

*.* @@logs.loggly.com:<port>

Note the two @@ signs! You can find more information in the rsyslog documentation.

It's not recommended, but if you need to use UDP, here's the configuration to forward data to your Loggly input:

*.* @logs.loggly.com:<port>

Configuring rsyslog for File Monitoring

The rsyslog server is installed by default on a number of distros. It has the capability to monitor a given file and send changes into syslog. If the rsyslog server is configured for it, these changes can also be forwarded to a remote central logging facility such as Loggly.

To enable log monitoring on a server, make sure it's running rsyslog and that rsyslogd has proper permissions to dynamically monitor your logs. "$PrivDropToGroup syslog" within rsyslog.conf should be "$PrivDropToGroup adm".

 

#What you need to change:
$PrivDropToGroup syslog adm
 

Once rsyslog is running, add the following lines to the the rsyslog.conf file (usually in /etc/):

#Only needs to be loaded once, like most rsyslog modules
$ModLoad imfile

#path to the file which you want to monitor
$InputFileName /var/log/apache2/access.log

#The tag apache can be changed to whatever you'd like *note this is referenced in the if $programname
$InputFileTag apache:

#the name of file within rsyslogs working directory
$InputFileStateFile stat-apache-access

#By default this is set to 'notice' options include standard syslog severity level of your choice
$InputFileSeverity info

#This is necessary for file monitoring (no parameters)
$InputRunFileMonitor

#Set to how often you'd like rsyslog to check out that file every second (defaults to 10 seconds)
$InputFilePollInterval 10 # This is a filter to the Loggly port number without a template if $programname == 'apache' then @@logs.loggly.com:<port>

The InputFileTag line tells rsyslog what to add as the tag in the log records. Choose whatever you like. Make it descriptive for the fie that you are monitoring. The InputFileStateFile is the file that will keep tap on how much of that file you have already sent in. Make this unique for each file that you are using.

Make sure that commonForwardFormat is a defined template in your rsyslog configuration otherwise remove it as rsyslogd would almost silently invalidate the line it's on.

The @@ in the last line of the configuration tells rsyslog to send data via TCP. Make sure you have a TCP input configured in Loggly. If you want to send the log files via UDP, use the following line.

*.* @logs.loggly.com:<port>

The complete documentation for the imfile plugin can be found at: imfile.

You can use imfile to monitor up to 100 files. Simply switch out the path to the file, make sure that the permissions are set:

$PrivDropToGroup syslog SHOULD BE → $PrivDropToGroup adm

The following is an example of an .conf file that can be placed within the rsyslog.d directory: Below is the rsyslog.conf which forwards the majority system messages to a Loggly input *as long as the apache.conf file is in the rsyslog.d directory apache2 access and error logs will go to their respective inputs.

Ubuntu Package

If you are running Ubuntu Maverick or Lucid, here is a PPA that you can add to install rsyslog (instead of using the old 4.2.0 version):

https://launchpad.net/~evax/+archive/rsyslog

WARNING: there is a bug with kernels < 2.6.34 causing rsyslog to eat cpu time

Make sure you have python-software-properties installed then add the repo, refresh apt and upgrade rsyslog.

sudo apt-get install python-software-properties &&\ sudo add-apt-repository ppa:evax/rsyslog &&\ sudo apt-get update &&\ sudo apt-get install rsyslog

Quick TLS

If you're using Ubuntu make sure that the following package is installed:

sudo apt-get install rsyslog-gnutls

Here's a config snippet for a TLS connection: 

$DefaultNetstreamDriverCAFile /path/to/loggly.com.crt
$ActionSendStreamDriver gtls
$ActionSendStreamDriverMode 1
$ActionSendStreamDriverAuthMode anon

*.* @@logs.loggly.com:<your_assigned_port>

Note that this ensures that the data travels encrypted but doesn't verify the identity of the peer. This configuration is thus potentially vulnerable to man-in-the-middle attacks.

The $DefaultNetstreamDriverCAFile parameter is required by the rsyslog gnutls module even if it's not used here.

TLS with peer checking

This is the most secure configuration as your data will be encrypted AND the Loggly server identity will be checked, but alas the most painful to set up.

You'll need this certificate, which is also provided on your TLS enabled input page (loggly.com.crt) and the intermediate certificate from Starfield called sf_bundle.crt.

Note the intermediary cert can be obtained by doing a wget on the box:

$ wget https://certs.starfieldtech.com/repository/sf_bundle.crt

Just to verify go ahead and double check the sha1sum or md5:

$ sha1sum sf_bundle.crt
9f4b50011bdeabda276c9dd08f32f545218ea1b7  sf_bundle.crt
$ md5sum sf_bundle.crt
f742e64a892167bb5b4a10da5a380425  sf_bundle.crt

The Sha1 is displayed on the same page where you can also obtain the intermediary cert via the browser -> Starfield. You then need to concatenate both:

$ cat {sf_bundle.crt,loggly.com.crt} > loggly_full.crt

Sadly, even if the server won't try to verify your identify you have to provide a dummy client certificate and key to rsyslog.

You can read on rsyslog's website how to generate a dummy certificate authority, and dummy certificates and keys for your client machines.

Now the configuration:

$DefaultNetstreamDriverCAFile /path/to/loggly_full.crt
$DefaultNetstreamDriverCertFile /path/to/dummy-cert.pem
$DefaultNetstreamDriverKeyFile /path/to/dummy-key.pem

$ActionSendStreamDriver gtls
$ActionSendStreamDriverMode 1
$ActionSendStreamDriverAuthMode x509/name
$ActionSendStreamDriverPermittedPeer @*.loggly.com@

*.* @@logs.loggly.com:<your_tls_enabled_input_port>

Note that rsyslog doesn't seem to handle wildcard certs properly so you can't write '$ActionSendStreamDriverPermittedPeer logs.loggly.com'

Dealing with Templates

You can create your own templates for logs like the following that forward syslog messages in JSON format (Hard coded template):

$template tpl, "{\"message\":\"%msg:::json%\",\"fromhost\":\"%HOSTNAME:::json%\",\"facility\":\"%syslogfacility-text%\",\"priority\":\"%syslogpriority-text%\",\"timereported\":\"%timereported:::date-rfc3339%\",\"timegenerated\":\"%timegenerated:::date-rfc3339%\"}"

Just remember that in order to enable a template for a forwarder you need to add it following the semi colon for example:

*.* @@logs.loggly.com:<port>;tpl #notice that the forward matches the $template name.

Bear in mind that the above does a very poor job handling escaped quotes in the %msg% property. With the off chance that you never have quotes in your message property the above works great. *This is taken care of in rsyslog v6 (Default JSON templates that don't barf the log on escaped quotes) More on v6 of rsyslog to come.

If you've already got JSON formatted logs just put the following in your config file:

$template json, "%msg%"

To use the template simply append it to your forwarder following a semicolon:

#TCP Input
*.* @@logs.loggly.com:<port>;json

#UDP Input
*.* @logs.loggly.com<port>;json

 

Top