Fixing Client IPs in Apache Logs with Amazon Load Balancers

Posted on 22. Mar, 2010 by Raffy in Code, Log Management

If you are running your Web servers behind a load balancer, you have probably noticed that your logs contain the load balancer’s IP address as the client IP, which is kind of annoying. There is an Apache module called RPAF, which fixes exactly this issue. Once you have it downloaded and installed, you configure it as follows:

RPAFenable On
RPAFsethostname On
RPAFproxy_ips 127.0.0.1 10.0.0.1
RPAFheader X-Forwarded-For

The module works such that it takes the IP address that is being transmitted in the X-FORWARD-FOR header and sticks it into the request as the ClientIP.

The not so nice part is that the RPAFproxy_ips statement lets you define the IP addresses of your load balancer. If you have only a set amount of them, all is fine. But here comes the catch. If you are deployed on Amazon AWS and you are using their load balancer (LB), then this solution is going to frustrate you quite a bit. You will notice that the LB’s IP address changes constantly and you keep adding IP addresses to the configuration statement. After about 10 IP addresses, I got sick of that and I started looking at the source code of RPAF to solve this problem once and for all. Here is what I did:

On line 139 of mod_rpaf-2.0.c, I added a return 1; statement. This will tell the is_in_array() function to always assume that the request is coming from a load balancer, without checking the configured list of IP addresses. The rest of the RPAF code is robust enough to only replace the client ip when an X-FORWARD-FOR header is actually set. After the change, do a make install-2.0 and you are in business.

Happy logging!

2 Responses to “Fixing Client IPs in Apache Logs with Amazon Load Balancers”

  1. Adam 23. Mar, 2010 at 10:53 #

    What happens if I send a spoofed X-FORWARD-FOR header? Can I write any entry in the logs that I want, thus masking my true IP during an attack?

  2. Raffy 23. Mar, 2010 at 11:01 #

    Good point. After doing some experimentation, I found that RPAF is doing the right thing. You can in fact change the X-FORWARDED-FOR header to this:

    X-FORWARDED-FOR: 1.1.1.1, 67.164.72.181

    Where 1.1.1.1 is the attacker. But as you can see, there are now two addresses in there and RPAF does the right thing of taking the last IP.