Syslog for newbies
When I was a kid, Saturdays were the days my dad worked on the car. He’d open the hood and look inside meaningfully. He already had an idea about what needed tuning or fixing because he wrote logs. He kept meticulous records of his gas mileage, oil changes, filter and hose replacements, etc. He knew the age, quality, and service expectations of parts in his car because he recorded all of that information.
Like I do with my shiny Linux box, my dad kept logs of his car’s performance. But while he wrote in an endless number of spiral-bound notebooks, my computer automatically gathers my log data. By default, Linux uses the syslog daemon to record logs about how the system is running, putting them into a central repository (/var/log) where I can examine them.
That’s all daemons like syslog, rsyslog, and syslog-ng really are: Services that log data about my system, posting that data to a spot where I can open the logs and take a look.
If you’re new to administering your own system, it’s good to know about these different daemons. You might decide that default syslog serves you just fine, or maybe rsyslog or syslog-ng with their more advanced functionality are better choices.
Syslog is the original. Its name alone can be a hurdle to the new administrator because there’s syslog the daemon, syslog the protocol, and syslog the message. The syslog daemon listens for messages on the syslog socket and then creates an entry in the appropriate log file. It uses syslog format for the message fields and data. Here’s an example syslog message about a failed login attempt:
<34>1 2003-10-11T22:14:15.003Z server1.com sshd – – pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.0.2.2
You can learn more about syslog message fields in this article: Linux Logging Basics.
Syslog creates log files that are technically readable. You just have to know what all the fields represent and the significance of their values. Frankly, unless you want the cred of wielding the tools of our ancestors on your Linux systems, I encourage you to look at rsyslog instead.
Rsyslog builds upon syslog, giving it flexibility. You configure it in a language called RainerScript, and one of the first things you likely want to do is to tell rsyslog where to send your log data. In this example, we’re sending our log data to a server called BEBOP.
action(type=”omfwd” protocol=”tcp” target=”BEBOP” port=”514″)
Since your system probably came with rsyslog already loaded and configured, directing log data to a particular server might be all the configuration you need to do.
Until your log data disk gets full, that is. Which is why you might want to set up a cron job for logrotate, which truncates aging log files and archives them. Without log rotation, rsyslog continues to collect log data, truncating it, overwriting older data, and if everything goes wrong, even crashing your system as the disk gets full. Anarchy!
Alternately, you can configure rsyslog to send all of your log data to a log management solution like Loggly, which archives your data and presents it to you in a way that you can easily search and analyze. Not anarchy!
In between syslog and rsyslog, syslog-ng came along to add more functionality to the syslog daemon. It’s got some great filtering capabilities. It can filter log messages according to their content, not just on fields like timestamp. When I’m looking through thousands of log messages, being able to zero in my search on specific content means the difference between solving my problem and calling my system unkind names.
Syslog-ng automatically processes log messages into a central repository and applies complex filtering to them. Syslog-ng can even parse your log data into JSON messages, which you can use in a log management tool like Loggly. Check out how easy it is to see at a glance who’s been logging in to your system and how often! So cool!
Coming back to my little Linux machine, syslog-ng has more than I use. This system is where I write, storing my work in the cloud. I’m not managing data centers or even multiple users. Just my own equivalent of Dad’s car, tuning it now and then, fixing it when it’s got a bit of a breakdown. I use rsyslog because that’s what’s on my system. I’m not a hobbyist nor a sysadmin, so “because that’s what’s on my system” is a good choice for me. Your own needs are likely different.