[ << | Thread Index | >> ]    [ << | Date Index | >> ]

Subject: Re: NAT through CIPE (not CIPE through NAT)
From: "E. Jay Berkenbilt" <ejb,AT,ql,DOT,org>
Date: Sun, 1 Jul 2001 15:51:50 +0200

SUMMARY: iptables SNAT rule seems to cause source address of encrypted
public cipe packets themselves to be altered; not just the packets
routed through ciped.  This must be either a bug in netfilter or a bug
in the way cipe interacts with netfilter.  This does not happen with
ipchains, even with the 2.4.3 kernel.  Using ipchains, NAT
(masquerading) through CIPE works.


I have more information about the message I sent out last weekend,
which is attached below for reference.  I'm hoping someone who
understands netfilter or cipe's interaction with it deeply can kick
in.  I believe that what we have here is either a bug in the netfilter
code or in the manner in which cipe interacts with it.  I have a lot
of evidence to support this:

 1.  Even with the 2.4 kernel, if I make sure that all iptables
     modules are unloaded and use ipchains to set up masquerading,
     then NAT through cipe works.  In other words, if I issue this

     ipchains -A forward -d -j MASQ

     then I'm in business.  However, I'd much rather use iptables than
     ipchains.  (Note: iptables with MASQUERADE target rather than
     SNAT target also fails.)

 2.  Using various LOG and other targets with iptables, it appears
     that the cipe packets are not passing through the tables as I
     would expect.  Maybe this is because my expectations are wrong.
     This does not happen with ipchains even with the 2.4 kernel.  For
     example, this ipchains command:

     ipchains -A output -p udp -d <public IP of site2-gw> 9999 -j REJECT

     stops cipe dead in its tracks exactly as expected.  However,
     these iptables commands:

     iptables -t filter -A OUTPUT -p udp -d <public IP of site2-gw> --dport 
9999 -j DROP
     iptables -t filter -A FORWARD -p udp -d <public IP of site2-gw> --dport 
9999 -j DROP

     have no impact.  (Recall that in ipchains, forwarded packets
     traverse both the forward and output chains, whereas in iptables,
     forward packets do not traverse the OUTPUT chain.)

     Also, if I use the LOG target in iptables to look at packets
     destined for site2-gw's public IP address on udp port 9999, I
     don't see any.  I can put these LOG targets in the nat filter's
     OUTPUT or POSTROUTING chains and in the filter table's OUTPUT or
     FORWARD chains, and I see nothing.  Actually, if I unload all
     modules and start everything from scratch, I get to see one
     single UDP packet logged in this way.  tcpdump shows the packets
     going out though.

 3.  If I use "tcpdump icmp or udp port 9999" and ping through my
     forwarded connection with no NAT enabled, I see icmp packets from
     my site1 internal address to my site2 internal address (as
     expected) and udp port 9999 cipe packets from my site1 external
     address to my site2 external address as expected.  However, once
     I enable SNAT, I see that the source address of the icmp packets
     are modified to the SNAT address as expected, but also, the
     source address of the UDP packets that are destined to site2's
     cipe daemon are also modified!  This means that the system is
     sending cipe packets with the source address and the
     destination address of site2-gw's public IP address.  There's no
     way this could ever work as there is no public route to  In fact, tcpdump on each gateway shows that the
     packets are going out but not being received on the other end.
     They are probably being blocked by some intermediate router.  (If
     I adjust my firewall rules appropriately, I can see them going
     out at the border of site1's network, but I can't see them coming
     in at the border of site2's network.  Ordinarily I don't allow
     192.168.* out un-NAT-ted anyway, so the packets would ordinarily
     never leave my network.)

I've tried all this stuff with the latest released cipe (1.5.2).  (The
development snapshot link on the cipe webpage is broken.)  I'm going
to investigate what changed between 2.4.3 (which I am now running
after installing the 2.4.3 kernel rpm from redhat's update area) and
2.4.5.  Any tips, including how to reach the correct audience for
additional help, would be helpful.  I think I've gone as far as I can
go without digging into the guts of ipip and netfilter in the kernel
(which will probably be my next step if no one swoops in with an
answer or a patch).




NAT through CIPE (not CIPE through NAT)


"E. Jay Berkenbilt" <ejb,AT,ql,DOT,org>


Sun, 24 Jun 2001 19:03:22 -0400

SUMMARY: IP forwarding across a CIPE VPN is working, but NAT across
the same CIPE VPN is failing.  tcpdump shows packets only on one side
of the interface.

Note: this question pertains to running NAT over CIPE, not to running
CIPE over NAT.  In other words, I have a working CIPE VPN between two
specific machines.  Each machine is on a private network.  I'd like to
talk between the two private networks, but one side doesn't have a
route to the other.  I am successful in routing between the two
networks using the CIPE boxes as gateways if I establish all the
required routing, but not in doing NAT over the CIPE interface.

Here are the details:

site1-machine: eth0:

site1-gateway: eth0:
               eth1: (dynamic public address)

site2-gateway: eth0:
               eth1: (static public address)

site2-machine: eth0:

All machines are running RedHat Linux 7.1 with cipe 1.4.6 as
distributed in RedHat 7.1 and with the default RedHat 7.1 2.4.2-based
kernel.  I've checked 1.5.2 but not installed it as it doesn't seem
that any changes are relevant to this problem.

site1-machine has a route for to site1-gateway.
site1-gateway has a route to through interface cipcb0.
site2-machine has site2-gateway as its default gateway.

site1-gateway has IP forwarding enabled and accepts forwarding from to any destination.

site2-gateway has IP forwarding enabled and accepts forwarding from to any destination.

site1's options file:

peer    (site2's public address):9999
key     (key)

site2's options file

me      (site 2's public address):9999
key     (key)

What works:

site1-gateway and site2-gateway can ping each other.  site2-gateway
sees the source address as  site1-gateway can ping
either or

site1-gateway and site2-machine can both ping each other since
site1-gateway knows that site2-machine is on the other side of the
CIPE VPN and site2-machine routes all non-local packets through
site2-gateway.  site2-machine can see but not, which is fine.

In order to get site1-machine and site2-machine to see each other, I
should be able to tell site1-gateway to NAT any packets being
forwarded to to source address  This does
not work.  I know, however, that I can forward packets through this
VPN without NAT.  Here are the details:

If I teach site2-gateway about with

route add -net dev cipbc0
iptables -t nat -I POSTROUTING -d -j ACCEPT
iptables -t filter -I FORWARD -d -j ACCEPT

then site1-machine and site2-machine can ping each other.
Furthermore, if I run tcpdump -i cipcb0 on both site1-gateway and
site2-gateway, I can see both the echo request and echo reply packets,
and I can see and as the source/destination
addresses.  This is exactly as expected.  Everything works perfectly.
My two networks can talk to each other.

However, I don't want site2 to know about  I want
site1-gateway to SNAT all its traffic to  This should be
easy.  Once the above situation works fine, I should simply need to
run the following on site1-gateway:

iptables -t nat -I POSTROUTING -d -j SNAT --to-source
and everything should just work.  (Note that site2-machine can ping fine.)  However, when I give this command, my tcpdump on
site1-gateway shows the echo requests with the source of
and the destination of as expected, but site2-gateway's
tcpdump shows nothing!

In other words, CIPE does not appear to be forwarding the traffic at
all.  tcpdump on site1 shows the packets being sent, but tcpdump on
site2 does not show the packets being received.

The thing that's baffling to me is that when I turn SNAT to the
site1's CIPE ip address, the cipe interface on site2 no longer appears
to be receiving packets even though the interface on site1 appears to
sending them.  Running strace on the ciped-cb processes is
unenlightening.  Any further tips on diagnosing this will be helpful.

I have administrative control of all machines in question, and I am
the only person using this VPN at the moment.  I have full freedom to
bring things up and down as required, so I can try experiments that
people may suggest.  One thing I have tried is to explicitly specify
both the peer: and me: parameters as static addresses (using the
address I happen to have now) on both sides.  This changes nothing --
I get exactly the same results.  When I try to NAT through the cipe
interface, tcpdump shows the packets on one side but not on the other.

For what it's worth, I used to use ppp over stunnel with otherwise
identical configurations.  NAT across that VPN worked fine.

E. Jay Berkenbilt <ejb,AT,ql,DOT,org>

[ << | Thread Index | >> ]    [ << | Date Index | >> ]