Ruby Forum NGINX > realip module broken?

We have been using (and relying) on the realip module for a while now to
extract 'X-Real-IP' headers from our loadbalancers so our backends get
REMOTE_ADDR variables. Everything seemed to work fine.

However today I was tracing some of my own debugging on a live server
(lots of rps) and was surprised by the output I got. Because I was doing
HEAD requests and all the other requests are GET's, a simple grep was
enough to filter out my debugging requests from the access_log

www.mydomain.com 200.199.140.205 - - [12/Aug/2008:09:57:04 -0400] HEAD /
HTTP/1.1 "403" 0 "-" "curl/7.16.3 (i686-pc-cygwin) libcurl/7.16.3
OpenSSL/0.9.8g zlib/1.2.3 libssh2/0.15-CVS"
www.mydomain.com 200.199.140.205 - - [12/Aug/2008:09:57:06 -0400] HEAD /
HTTP/1.1 "403" 0 "-" "curl/7.16.3 (i686-pc-cygwin) libcurl/7.16.3
OpenSSL/0.9.8g zlib/1.2.3 libssh2/0.15-CVS"
www.mydomain.com 88.80.254.219 - - [12/Aug/2008:09:57:10 -0400] HEAD /
HTTP/1.1 "403" 0 "-" "curl/7.16.3 (i686-pc-cygwin) libcurl/7.16.3
OpenSSL/0.9.8g zlib/1.2.3 libssh2/0.15-CVS"
www.mydomain.com 88.80.254.219 - - [12/Aug/2008:09:57:12 -0400] HEAD /
HTTP/1.1 "403" 0 "-" "curl/7.16.3 (i686-pc-cygwin) libcurl/7.16.3
OpenSSL/0.9.8g zlib/1.2.3 libssh2/0.15-CVS"
www.mydomain.com 66.153.141.107 - - [12/Aug/2008:09:57:55 -0400] HEAD /
HTTP/1.1 "403" 0 "-" "curl/7.16.3 (i686-pc-cygwin) libcurl/7.16.3
OpenSSL/0.9.8g zlib/1.2.3 libssh2/0.15-CVS"
www.mydomain.com 72.187.80.90 - - [12/Aug/2008:09:58:16 -0400] HEAD /
HTTP/1.1 "403" 0 "-" "curl/7.16.3 (i686-pc-cygwin) libcurl/7.16.3
OpenSSL/0.9.8g zlib/1.2.3 libssh2/0.15-CVS"
www.mydomain.com 72.187.80.90 - - [12/Aug/2008:09:58:30 -0400] HEAD /
HTTP/1.1 "403" 0 "-" "curl/7.16.3 (i686-pc-cygwin) libcurl/7.16.3
OpenSSL/0.9.8g zlib/1.2.3 libssh2/0.15-CVS"


But hold on... Why am I getting al those random source IP's?

After confirming (using a strace) that the correct headers are being
sent to Nginx, the only conclusion I can draw is that the realip module
is mixing up IP adresses. Zipping throught the logs seems to confirm
this, when I do a request I either see my own IP address (correct) or
the IP address from the request right before me (incorrect).

Does this ring a bell for anyone? It seems to me like the realip module
is seriously broken (at least in 0.6.32). I had a look at the source
code, but cannot come with anything obvious.

Spil Games wrote:
> Does this ring a bell for anyone? It seems to me like the realip module
> is seriously broken (at least in 0.6.32). I had a look at the source
> code, but cannot come with anything obvious.

I think I figured it out. We run multiple Nginx backends behind a Zeus 
ZXTM loadbalancer. The loadbalancer is configured to use keepalives to 
the backends and thus pipes requests from various source addresses 
through one connection.

When the first request on a new lb->ws connection comes in, Nginx 
matches the source address to the value set through 'set_real_ip_from'. 
Let's assume the request originates from the loadbalancer (obviously :P) 
and that we have set 'set_real_ip_from' appropriately. The realip module 
will patch the following:

  sin->sin_addr.s_addr = addr;
  r->connection->addr_text.len = len;
  r->connection->addr_text.data = p;

So the IP address is corrected for this connection. So far, so good.

Now to the second request: Nginx will try to match the source address 
again, but because this is the same (already patched) connection, it 
will not match 'set_real_ip_from' and the address will pass unmodified, 
which is incorrect because this request originates from a different 
client ip address.

I can work around this problem by setting 'set_real_ip_from' to 
'0.0.0.0/0', but I think this is essentially a bug in the module. It 
should not patch the source ip address for the whole (keepalive) 
connection, but only for the current request.

On Tue, Aug 12, 2008 at 04:07:39PM +0200, Spil Games wrote:

> HTTP/1.1 "403" 0 "-" "curl/7.16.3 (i686-pc-cygwin) libcurl/7.16.3
> www.mydomain.com 66.153.141.107 - - [12/Aug/2008:09:57:55 -0400] HEAD /
> But hold on... Why am I getting al those random source IP's?
> 
> After confirming (using a strace) that the correct headers are being
> sent to Nginx, the only conclusion I can draw is that the realip module
> is mixing up IP adresses. Zipping throught the logs seems to confirm
> this, when I do a request I either see my own IP address (correct) or
> the IP address from the request right before me (incorrect).
> 
> Does this ring a bell for anyone? It seems to me like the realip module
> is seriously broken (at least in 0.6.32). I had a look at the source
> code, but cannot come with anything obvious.

What loadbalancer (that sets X-Real-IP) do you use ?

Igor Sysoev wrote:
> What loadbalancer (that sets X-Real-IP) do you use ?

Our posts crossed eachother, I think. Please see my previous mail. We're 
running Zeus ZXTM. I have implemented the mentioned workaround now, and 
it seems to work. I have no clue however how to repair the bug in the 
module. It probably requires saving the original source address 
somewhere.

On Tue, Aug 12, 2008 at 04:57:10PM +0200, Spil Games wrote:

> When the first request on a new lb->ws connection comes in, Nginx 
> 
> Now to the second request: Nginx will try to match the source address 
> again, but because this is the same (already patched) connection, it 
> will not match 'set_real_ip_from' and the address will pass unmodified, 
> which is incorrect because this request originates from a different 
> client ip address.
> 
> I can work around this problem by setting 'set_real_ip_from' to 
> '0.0.0.0/0', but I think this is essentially a bug in the module. It 
> should not patch the source ip address for the whole (keepalive) 
> connection, but only for the current request.

I suspected those are loadbalancer keepalive connections.
I will look how to resolve the issue. Right now you may disable 
keepalive
on nginx side.

Igor Sysoev wrote:
> I will look how to resolve the issue. Right now you may disable 
> keepalive on nginx side.

I'll keep it at 0.0.0.0/0 for now. The 'set_real_ip_from' directive 
doesn't add much security anyway:

a) The loadbalancer overwrites any existing X-Real-IP headers.

b) Even if a) would not be done, the header would be accepted because 
all requests come from the loadbalancer IP.

It would be nice if the realip module could be fixed though. It's a 
matter of semantics, but I believe an X-Real-IP (or X-Forwarded-For) 
header should only influence the request, not the entire connection.

Thanks.

On Wed, Aug 13, 2008 at 10:23:27AM +0200, Spil Games wrote:

> all requests come from the loadbalancer IP.
> 
> It would be nice if the realip module could be fixed though. It's a 
> matter of semantics, but I believe an X-Real-IP (or X-Forwarded-For) 
> header should only influence the request, not the entire connection.

The attached patch should fix the issue.

Igor Sysoev wrote:
> The attached patch should fix the issue.

Thanks!

The patch did not apply cleanly to 0.6.32. Attached patch does.