Support HAProxy logging Using Syslog

HAProxy Logs

This document provides an overview of the features and benefits of using load balancing with HAProxy. Load balancing provides better performance, availability, and redundancy because it spreads work among many back-end servers. This will:

  • Reduce the load on each individual machine.
  • Let you do maintenance, upgrade software, deploy websites, or swap out servers without disrupting service.
  • Enable your website or service to handle more traffic.
  • Increase overall performance.

HAProxy is a free, open-source reverse proxy and load balancer with the ability to handle hundreds of thousands of simultaneous connections. It has several features which allow it to work well with web traffic, such as its ability to inspect and direct clients based on their HTTP messages.

1. Configure Syslog Daemon

Run our automatic Configure-Syslog script below to set up rsyslog. Alternatively, you can Manually Configure Rsyslog or Syslog-ng.

curl -O https://www.loggly.com/install/configure-linux.sh
sudo bash configure-linux.sh -a SUBDOMAIN -u USERNAME

Replace:

  • SUBDOMAIN: enter the account subdomain that you created when you signed up for Loggly
  • USERNAME: enter your Loggly username, which is visible at the top right of the Loggly console

You will need to enter your system root password so it can update your rsyslog configuration. It will then prompt for your Loggly password.

2. Install and Start HAProxy

Install the haproxy package with following command:

sudo apt-get install haproxy

After installation, verify that HAProxy is working:

haproxy -v

The server will respond with:

HA-Proxy version 1.6.3 2015/12/25
Copyright 2000-2015 Willy Tarreau <willy@haproxy.org>

You can also find created files in the following locations:

/etc/default/haproxy
/etc/haproxy
/etc/init.d
/etc/logrotate.d/haproxy
/run/haproxy
/usr/sbin
/usr/share/lintian/overrides/haproxy
/var/lib/haproxy

The HAProxy main configuration file is located at:

/etc/haproxy/haproxy.cfg

The above file starts a server with the default settings and below is the content of this file:

global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin
        stats timeout 30s
        user haproxy
        group haproxy
        daemon

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        # Default ciphers to use on SSL-enabled listening sockets.
        # For more information, see ciphers(1SSL). This list is from:
        #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
        ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
        ssl-default-bind-options no-sslv3

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

The global section contains settings that apply to the HAProxy process itself. This section contains user, group, log directives, and stats. The defaults section contains all of the proxies settings.

After editing /etc/haproxy/haproxy.cfg, restart the haproxy service for the changes to take effect:

sudo service haproxy restart

Step 3. Setting Up Required Configurations and Load Balancing

Now let’s put HAProxy in front of your web server(s).

At the bottom of the /etc/haproxy/haproxy.cfg file, we will add one or more new sections. In addition to global and defaults you can add:

  • frontend: defines a reverse proxy which will listen for incoming requests on a specific IP address and port.
  • backend: defines a pool of servers that the frontend will forward requests to.
  • listen: a shorthand notation that combines frontend and backend features into a single command.

Let’s define a frontend first. We will have it listen on the HAProxy IP address at port 80. Open /etc/haproxy/haproxy.cfg configuration file:

sudo vi /etc/haproxy/haproxy.cfg

Add the following lines to the bottom of the /etc/haproxy/haproxy.cfg file:

frontend firstbalance
        bind *:80
        option forwardfor
        default_backend webservers

You can use either an IP address or an asterisk *, which means any IP address configured on this machine. You can add as many bind directives to frontend as you want.

We also want to capture the client’s source IP address in our web server’s logs. We don’t want to log our proxy server’s IP address. To do this, we added a header called forwardfor to the incoming request before it is passed along to the web servers. Then the web server can look for and parse this header to get the original, client IP address.

We can also add an optional forwardfor directive to the defaults section so that it applies across the board to all proxies.

Our frontend section is configured, via its default_backend directive, to forward requests to a pool of servers defined in a backend section called webservers. We can distribute the load across many servers. We just add more server directives. webservers backend shares the traffic equally among two web servers by using the roundrobin algorithm.

The roundrobin load-balancing algorithm is the default if you don’t set the balance directive. A balance directive can be set in a backend section or globally in the defaults section. If it’s set in the defaults section, it can be overridden within a backend to use a different algorithm. webserver1 would get the first request. The second would go to webserver2. The third would go back to webserver1, etc. We can add as many server directives as we want to a single backend.

We enabled health checking by adding the check parameter to the webserver1 and webserver2 servers directive. A drawback of enabling health checks with the check parameter is that success is measured only by the ability to make a TCP connection to the back-end servers IP address and port. So, even if the web servers begins responding with HTTP 500 Internal Server Error, the checks will still pass as long as a connection can be made. So, visitors might be sent to web pages that are displaying errors. We are using mode http in default section. We can have them look for a successful HTTP response. Any response between the 2xx-3xx range is good. To use this feature, we added the option httpchk directive to the back-end.

Let’s go ahead and define that. Add the following lines below the frontend section:

backend webservers
        balance roundrobin
        server webserver1 Your-Webserver1-IP:80 check
        server webserver2 Your-Webserver2-IP:80 check
        option httpchk

Replace:

  • Your-Webserver1-IP: enter your own webserver IP

Save changes and close the /etc/haproxy/haproxy.cfg file.

We can verify that the configuration file is valid with the following command. If there are any errors, they will be displayed so that we can go in and fix them:

haproxy -f /etc/haproxy/haproxy.cfg -c

Now that the necessary changes have been made, restart the haproxy service:

sudo service haproxy restart

Step 4. Generate Events

Now you can create some HTML/PHP files on the web servers for testing the HAProxy load balancer. When you make a request to the load balancer’s IP address, you should see your HTML or PHP web content.

For example, if I hit the localhost URL, as shown below, it will generate and send the event to Loggly.

curl http://localhost:80/First

Step 5. Verify Events

Search Loggly for events with the RsyslogTLS tag over the past hour. It may take a few minutes to index the event. If it doesn’t work, see the troubleshooting section below.

Advanced HAProxy configurations Options

    • If you want, you can use a single listen section as an alternative. The listen section uses a single listen directive instead of using both a frontend and backend section. The listen section combines the features of both. To implement that, you should remove both the existing frontend and backend section from the /etc/haproxy/haproxy.cfg file and replace them with the configuration below at the bottom of the file:
      listen firstbalance
              bind *:80
              balance roundrobin
              option forwardfor
              option httpchk
              server webserver1 Your-Webserver1-IP:80 check
              server webserver2 Your-Webserver2-IP:80 check
    • If you would like to monitor live traffic that passes through HAProxy, enable debugging with the -d flag:
      haproxy -f /etc/haproxy/haproxy.cfg -d

HAProxy Logs Troubleshooting

If you don’t see any data show up in the verification step, then check for these common problems.

Thanks for the feedback! We'll use it to improve our support documentation.