Jails and loopback interfaces

I've got a server running FreeBSD 6.2 and PF. The server has a couple  
dozen jails on it. Previously, I had a few "private" services such as  
MySQL running on loopback IPs (127.0.0.2+) and the rest of the jails  
running on the public IPs.

I have to renumber my machine with a new block of public IPs so I  
thought I'd be clever and move all the jails onto loopback IPs. Then  
I could use PF to redirect the new IPs and the old ones to the  
corresponding jails.

The relevant parts of my PF config look like this:

> ext_if="em0"
> loop_if="lo0"
>
> ext_addr="72.29.111.130"
> loop_addr="127.0.0.101"
>
> ext_net ="72.29.111.128/27"
> ext_net2="208.75.180.64/27"
> loop_net="127.0.0.0/8"
>
> lo_simerson     = "127.0.0.4"
> lo_toaster      = "127.0.0.6"
> lo_tnpi         = "127.0.0.13"
>
> simerson     = "72.29.111.131"
> toaster      = "72.29.111.133"
> tnpi         = "72.29.111.140"
>
> # nat the jails running on loopback to their public ip
> nat on $ext_if from $lo_simerson     to any -> $simerson
> nat on $ext_if from $lo_toaster      to any -> $toaster
> nat on $ext_if from $lo_tnpi         to any -> $tnpi
>
> # specific jail rules
> rdr on $ext_if from any to $simerson     -> $lo_simerson
> rdr on $ext_if from any to $toaster      -> $lo_toaster
> rdr on $ext_if from any to $tnpi         -> $lo_tnpi

This works just great. Traffic from the outside world gets redirected  
to the correct jail via a rdr rule. The corresponding nat rule alters  
the source IP from each jail to its corresponding public IP.

The problem I have with this arrangement is when a jail attempts to  
connect to the public IP of another jails, the connection fails. So,  
a client running in one jail can't send email to my mail server  
running in another jail. This problem is described in detail here:

     http://www.openbsd.org/faq/pf/rdr.html

And I have attempted to configure the RDR and NAT combination  
solution as shown. But after trying a few dozen iterations, I can't  
get it to work. Based on my understand of the PF manual,  
conceptually, the problematic connection looks like this:

    simerson (127.0.0.4)  -> toaster public IP (72.29.111.133)  
rewritten to toaster private IP (127.0.0.6)
    toaster (127.0.0.6) -> simerson (127.0.0.4)

So the simerson jail gets returned a packet from the toasters  
loopback IP, which it ignores because it is not the IP it sent the  
connection attempt to.

But no matter what I try, I can't make it work. PF is not denying any  
connections. Without resorting to split-horizon DNS, has anyone  
gotten this to work reliably on FreeBSD?   I'd love to see a working  
configuration.

Matt

_______________________________________________
freebsd-security@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-security
To unsubscribe, send any mail to "freebsd-security-unsubscribe@freebsd.org"