Tuesday, November 15, 2011

Fun with fragments and brouters

I've been working on a project that involves transparently intercepting traffic from a Linux bridge and manipulating the traffic.  I've learned all kinds of things about trproxy, iptables, policy routing, and other exciting technologies.  This week's life lesson has involve IP fragmentation.

Tproxy requires that packets are routed to the loopback interface, so that packets not really addressed to the intercepting machine are delivered to a process on that machine (thus implementing the "transparent" proxy).  This means that my bridged machine must route some of the traffic.  No problem, the broute table in ebtables enables us to identify some traffic to move from the bridge path to the routed path.

In my case, the traffic I'm interested in is not well defined by a particular port or small set of ports.  I have to consider all high port traffic.  So I have the following rules in ebtables

ebtables -t broute -A BROUTING -p IPv4 --ip-protocol udp  --ip-source-port 1024:65535 --ip-destination-port 1024:65535 -j redirect --redirect-target DROP

ebtables -t broute -A BROUTING -p IPv4 --ip-protocol tcp  --ip-source-port 1024:65535 --ip-destination-port 1024:65535 -j redirect --redirect-target DROP

All was well with the world until we encountered an environment with SIP and TFTP.  Some packets appeared on one interface and didn't get passed to the other interface.  However ICMP messages about time exceeded did appear.  Then I noticed that the missing packets were all fragmented.  Finally, it hit me that the broute tables is evaluated before IP defragmentation is even considered.  So the first fragment contains the port information and gets passed to the routing path, but the subsequent fragments do not contain port information and so stay on the bridge path.  The other fragments never appear along the route path, so the first fragment is never forwarded.

Once I changed the ebtables rule to not check the ports but forward all TCP and UDP traffic to the routing path, the packets started flowing again.

It seems that with path MTU discovery it is fairly rare to find IP fragments of TCP streams these days, but with UDP traffic it is the responsibility of the application writer to create small enough packets to avoid fragmentation.  Thus, I didn't see this problem until we had UDP based SIP and TFTP services in the environment.

No comments:

Post a Comment