Scrub Sensitive Data in Rsyslog

When your company has data that it should not expose due to concerns about security or privacy, you can scrub or mask the data from the logs. This removes the sensitive information before it leaves your network. Alternatively, we offer role based access control if there is sensitive information you prefer to keep in the logs, and you want to control who has access. The following example will scrub a 16 Digit Credit Card Number from the logs, but you can scrub any string that follows a regular expression pattern. This example requires Rsyslog version 8.x or higher.

Rsyslog Setup

  1. Configure Syslog Daemon
  2. If you haven’t already, run our automatic Configure-Syslog script below to setup rsyslog. Alternatively, you can Manually Configure Rsyslog or Syslog-ng.

    curl -O https://www.loggly.com/install/configure-linux.sh
    sudo bash configure-linux.sh -a SUBDOMAIN -u USERNAME
    

    Replace:

    • SUBDOMAIN: your account subdomain that you created when you signed up for Loggly
    • USERNAME: your Loggly username, which is visible at the top right of the Loggly console

    You will need to enter your system root password so it can update your rsyslog configuration. It will then prompt for your Loggly password.

  3. Update Configuration
  4. Copy and replace the below mentioned LogglyFormat Template in /etc/rsyslog.d/22-loggly.conf

    $template LogglyFormat,"<%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [TOKEN@41058] %$!msg%\n" 

    Replace:

    Add the following code below LogglyFormat Template in same file. This example will match 16 Digit Credit Card number. Please substitute your own regular expression to meet your own requirements.

    if re_match($msg,'(5[1-5][0-9]{14})') 	
    then 
    {
        set $!ext = re_extract($msg,'(5[1-5][0-9]{14})',0,1,"");
        set $!msg= replace($msg, $!ext, "xxxxxxxxxxxxxxxx");
    }	
    else 
        set $!msg = $msg;
    

    Save the file and restart rsyslog.

     sudo service rsyslog restart

  5. Send A Test Event
  6. Use Logger to send a test event to Loggly. In the example below we are sending one sample credit card number.

    logger 'credit card number is 5255224165541111'
    

  7. Verify
  8. Search Loggly over the past 10 minutes to find your logs. It may take a few minutes to index them. Click on one of the logs to show a list of syslog fields along with scrubbed info. If you don’t see them, check the troubleshooting section below.
    Scrubbed log scrub

Advanced Rsyslog Configuration Options

Troubleshooting Linux Syslog

  • The rsyslog versions supported for this example are 8.x or higher as CEE Lumberjack properties are not supported by earlier versions.
  • Make change in template where we are declaring a new variable %$!msg%
  • The regex which you are using in the script could be invalid rsyslog regex. Please test the regex and input string.
  • Try manually configuring rsyslog if the script doesn’t work
  • See our Rsyslog Troubleshooting Guide
  • Search or post your own question in the community forum.
Thanks for the feedback! We'll use it to improve our support documentation.


Top