Detalied analysis of UDP recv refused and query about fixing it
Len Reed <lreed,AT,linuxcare,DOT,com>
Tue, 09 Dec 2003 12:57:37 -0500
I'm seeing UDP connection refused. I see complaints about this in the
mailing list archives, but I couldn't find a detailed analysis or fix.
I'm posting my analysis here; please let me know if there's a fix or
you'd like to help me craft one. I know network programming but am
unfamiliar with cipe internals.
The problem is that, after pkcipe successfully negotiates the
"connection", the server end refuses to accept UDP datagrams.
From the client's syslog:
ciped-cb: kxchg: recv: Connection refused
My setup is:
0. Both ends running Linux, cipe 1.5.4.
1. pkcipe client behind a gateway that is doing NAT. (In my case the
client is single-homed if you don't count the cipe point-to-point.)
2. pkcipe server is multi-homed. The outward-facing NIC handles the
Internet traffic; an inward-facing NIC routes traffic to/from the
private 10.*.*.* networks.
Everything below I learned from netstat, route, tcpdump, etc., plus the
cipe config files. I haven't dug into the source code yet.
If the client's eth0 address is 192.168.x.y, it *always* works. If the
client's eth0 address is 10.0.0.z, it almost never works. (Open
mystery: "why does it work every once in a while?")
The failure mode is that pkcipe negotiates fine, but the server refuses
the UDP packets, returning ICMP connection refused. Running nestat
shows that problem is that the cipe process that should be listening for
those UDP packets is bound to the wrong interface! It's bound to the
inward-facing interface! When the datagrams show up on the correct
interface, they get rejected.
It appears that the server is using the client's local, unNATted IP
address when binding the UDP port. If that address is in 10.0.0.0/8, it
gets bound to the inward facing (wrong) LAN. If the client is in
192.168.0.0/16, though, it gets bound correctly. (I presume that this
is, too, is luck rather than proper programming; with no explicit route
to the client, it binds to the default route, which happens to be the
A supporting clue can be found in /var/run/cipe/<user> on the server.
This contains the IP address of the client. Instead of the public
Internet address, which is what I'd expect to find there, it contains
the private unNATted address. I can see no reason for the server to
know this address or care about it. (In fact, in the general case, a
machine doesn't have a single address. If the client host had three
NICs, which IP address would be chosen? The one that routes to the NAT
interface? Beats me. IMO all answers are wrong. These addresses
should be hidden from the server.) Furthermore, for it to even know the
private IP address, that address had to be passed as data during the
pkcipe negotiation; the IP headers of course contain the NATted (public)
So, solutions could include
1. Accepting UDP on any interface. Since there are session keys in
force, I don't see that this would be a security hole. It doesn't seem
quite as nice as #2.
2. Fix it to bind to the correct interface. That interface is the one
that pkcipe connection came in on.
An alternate #2 might involve making /var/run/cipe/<user> contain the
public, not private address of the client. Correct behavior might just
"fall out" then. (As I said, using the private address just seems
wrong.) There may be compatibility implications w.r.t. existing cipe
clients if doing this.
Let me know if there's a solution or workaround for this; I need to fix
this in the next couple of days so I'll be digging into the code soon
Oh, one last note. Someone reading this is going to mistakenly think
that the routes overlap and so this is in an unsolveable problem. Not
so. Technically, the inward facing 10.0.0.0/8 does overlap the
10.0.0.0/24 that my hotel put me on. (Now you see why I want to solve
this now; it works fine from home, where I use 192.168, but the hotel
isn't giving me that choice.) However, I don't need to get to anything
in the 10.0.0.0/24 range behind the server, so it doesn't matter. In
fact, in the worst case, I ought to be able to set up my laptop's
routing so that I can get to anything in 10.0.0.0/8 at the other end
except (a) the address my laptop is using on the hotel subnet and (b)
the default gateway on that subnet. (There's nothing else on the hotel
LAN I care about; I'm too busy to try to hack into other hotel guests'
laptops. :)) Dealing with (a) or (b) would be very ugly; maybe I could
use some iptables magic to deal with that.