Gen1: Logging with syslog-ng

[obsolete]

Table of Contents

We highly recommend using syslog-ng as your syslog daemon. syslog-ng OSE is an open source alternative to the standard syslog daemon that's commonly found on UNIX and UNIX-like (*nix) systems. It uses the basic syslog protocol, but extends it with content-based filtering, flexible configuration options and adds important features, such as using TCP (as well as TLS), which is much more reliable than UDP.   syslog-ng OSE is developed by Balabit  If you'd like some additional information on syslog-ng head over to Wikipedia or Balabit's overview of syslog-ng OSE.

You can use syslog-ng to monitor log files on your servers and forward them to Loggly. A bare bones configuration to forward local syslog events and monitor a file located at /path/to/your/file and send it to an input on Loggly listening to port 1234 looks something like this:

source s_all {
internal();
unix-stream("/dev/log");
file("/path/to/your/file" follow_freq(1) flags(no-parse));
};
destination d_loggly {
tcp("logs.loggly.com" port(1234));
};
log {
source(s_all); destination(d_loggly);
};

syslog-ng Versions

There are numerous versions of syslog-ng available for download and packaged for use on different distributions. If your distro already has syslog-ng packaged up on it, you should double check which version you're running:

$ /opt/syslog-ng/sbin/syslog-ng -V
syslog-ng 3.0.5 <--- this is out of date

Loggly recommends running the latest version of syslog-ng if you will be monitoring a set of standalone log files. To find the latest, visit Balabit's download page. Older versions of syslog-ng may not have support for forwarding to custom TCP ports, or monitoring standalone log files.  Know the limitations of your version by going through Balabit's docs for your version of syslog-ng OSE.

syslog-ng Documentation Links

 

Getting syslog-ng Up and Logging

While many distributions have syslog-ng available as a package, most default distros aren't using the current version of syslog-ng. It is likely you'll need to do a manual update of syslog-ng to the newest version.

These instructions assume you are running a Debian flavored distro, but should be easily adaptable to other OS builds.

Uninstall Older Versions

If you are running another syslog daemon (including an older version of syslog-ng) which was installed by a package manager such as aptitude or yum, Loggly recommends uninstalling it before you begin installing the newer version of syslog-ng.

Here's an example of uninstalling syslog-ng with aptitude on a Debian box:

$ aptitude remove syslog-ng -y

Download, Install and Launching

All of the syslog-ng versions are available for download on Balabit's website. The source code is also available for building by hand, but we won't cover that here.

If you have a fairly common distribution, you can browse to the binary build directory (for 3.1.4) and download your package. Pick the build best suited for your distro, and then download it (assuming a 64-bit Debian package):

$ wget "http://www.balabit.com/downloads/files?path=/syslog-ng/open-source-edition/3.1.2/setups/debian-etch-amd64/syslog-ng_3.1.2_amd64.deb"

Still assuming you have a Debian distro, you can do the following to install syslog-ng:

$ dpkg -i syslog-ng_3.1.2_amd64.deb

Note: Many Debian distributions require some type of syslog service be installed and running. If you run into dependency issues, try uninstalling whatever depends on syslogd first, then install syslog-ng via the binary package and reinstall those packages.

Assuming your installation succeeded, it should have started syslog-ng for you:

Setting up syslog-ng (3.1.2) ...
Restarting syslog-ng: Stopping syslog-ng: Starting syslog-ng: *

You can also start, stop and restart syslog-ng using the init.d script:

$ /etc/init.d/syslog-ng restart

Configuration

The syslog-ng configuration file should be located in the /opt/syslog-ng/etc/ directory if you installed one of the binary packages:

/opt/syslog-ng/etc/syslog-ng.conf

Edit that file and make sure you have a source called s_all that looks something like this:

source s_all {
internal();
unix-stream("/dev/log");
file("/proc/kmsg" program_override("kernel: "));
};

Forwarding data to another syslog server (like Loggly) requires setting a destination directive which tells syslog-ng where to forward the data it collects. You can reference the name, protocol and port destinations by going to the Input Management page in your Loggly account:

gen1_input_management

Now take the protocol and port, and put them in a destination entry (this example uses the tcp protocol):

destination d_loggly {
tcp("logs.loggly.com" port(10997));
};

Be sure to replace the 10997 above with the port number that is shown on your input page! After you've added that line, put in the log line that tell syslog-ng to forward Loggly the s_all source to the d_loggly destination:

log {
source(s_all); destination(d_loggly);
};

All together, your config should look something like this:

source s_all {
internal();
unix-stream("/dev/log");
file("/proc/kmsg" program_override("kernel: "));
};
destination d_loggly {
tcp("logs.loggly.com" port(10997));
};
log {
source(s_all); destination(d_loggly);
};

Once you are done configuring syslog-ng, restart it:

$ /etc/init.d/syslog-ng restart

Make sure you've turned on discovery mode on the input you are using on Loggly. Discovery mode is enabled on an input by clicking on the slider switch next to the input on the Input Management page. Once you have discovery mode turned on for the input, send some data through syslog-ng to have it forwarded to your Loggly account:

$ logger "loggly is better than a bee in your aunt's bonnet"

Give the beavers that run the indexers on Loggly a few moments, then do a search in the shell for part of the string you just sent in:

admin@demo> search aunt bee

Quick TLS Setup

Needless to say, there are going to be many times when you're going to want your logs encrypted during transport. This is where TLS comes in. The quick setup will ensure that your logs go to Loggly encrypted, but it will skip the step where Loggly validates *you* (which prevents man-in-the-middle attacks).

 

Place this in your config (/etc/syslog-ng.conf):

destination d_syslog_tls {
tcp("logs.loggly.com" port(33267)
tls(peer-verify(required-untrusted)
ca_dir('/opt/syslog-ng/keys/ca.d/') ));
};
log { source(s_all); destination(d_syslog_tls); };

You'll need this certificate, which is also provided on your TLS enabled input page (loggly.com.crt) and the intermediate certificate from Starfield, called sf_bundle.crt.

Note the intermediary cert can be obtained by doing a wget on the box:

$ wget https://certs.starfieldtech.com/repository/sf_bundle.crt

Just to verify go ahead and double check the sha1sum or md5:

$ sha1sum sf_bundle.crt
9f4b50011bdeabda276c9dd08f32f545218ea1b7  sf_bundle.crt
$ md5sum sf_bundle.crt
f742e64a892167bb5b4a10da5a380425  sf_bundle.crt

The Sha1 is displayed on the same page where you can also obtain the intermediary cert via the browser. You then need to concatenate both:

$ cat {sf_bundle.crt,loggly.com.crt} > loggly_full.crt

Of course, you'll need to restart syslog-ng to see your changes take effect – for TLS, you may want to start syslog-ng with the -d flag (for debug) so you can get an idea of what's happening – TLS can be a little tricky to get just right.

 

Monitoring a File

Syslog servers handle syslog events from any services that support logging to the syslog handler on a given box. By default, some services do not log into the local syslog server. To get logs from services like Apache to Loggly, you'll need to tell syslog-ng to monitor the log files that the services generates.

Let's take a simple example where you need to monitor the access.log and error.log files in the /var/log/apache2/ directory on a Debian box. The instructions for monitoring those files with syslog-ng should be placed in the source directive:

file("/var/log/apache2/access.log" follow_freq(1) flags(no-parse));
file("/var/log/apache2/error.log" follow_freq(1) flags(no-parse));

If you apply that to the configuration above, you get something that looks like this:

source s_all {
internal();
unix-stream("/dev/log");
file("/proc/kmsg" program_override("kernel: "));
file("/var/log/apache2/access.log" follow_freq(1) flags(no-parse));
file("/var/log/apache2/error.log" follow_freq(1) flags(no-parse));
};
destination d_loggly {
tcp("logs.loggly.com" port(10997));
};
log {
source(s_all); destination(d_loggly);
};

Restart syslog-ng to have the changes take effect, and then hit Apache to generate some events. Jump over into the shell on Loggly and do a search for events with a 200 in them:

admin@demo> search 200
- - [19/Oct/2010:17:22:20 -0700] "GET /rooster/ HTTP/1.1" 200 1516 "http://archives.geekceo.com/" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_4; en-US) AppleWebKit/534.3 (KHTML, like Gecko) Chrome/6.0.472.63 Safari/534.3"

You can get more information about other syslog based solutions on the Logging Configuration page.

Top