Thursday, November 17, 2011

Another Learning Experience Being in the Middle of the Network

As I noted in my earlier blog, I've been working on a project that involves intercepting network packets on a bridged/routed linux box transparently.  I'm intercepting more packets than I really need, and processing the first few bytes of the session to determine if the packet is part of a protocol I care about.  If I determine this isn't the target protocol, the program mindlessly forwards on the rest of the packets in the session.

This strategy seemed to work reasonably well.  The program has been in various states of test for about a year and seemed to mostly work as advertised.  Finally got paired with a more dedicated engineer from the customer company to finish up the testing, and we deployed in the office environment.  Slingbox from the office stopped working.  Read up on the slingbox protocol, took packet captures from both interfaces, and stared at them.

Finally, I noticed that the first packet of the TCP session on the incoming interface was N bytes, but two packets of size O and P (such that O + P = N) appeared on the outgoing interface.  My program passing along the the for O bytes before deciding it wasn't the write protocol and passing along the rest of the packet.

The slingbox program "should" have been smart enough to keep ready if it didn't get enough bytes on the first read, but evidently this particular slingbox implementation didn't.  So as the intermediary program, my program adapted and buffered its write, and then all was well with slingbox.

I noticed a few days later that I was getting fewer computers communicating along the desired protocol with my program in the middle than when my program was out of the picture.  Saw that I was splitting the first packet for the protocol match case too.  Evidently some implementations of the target protocol also don't continue reading on the TCP socket.  I fixed it to buffer the first packet in all cases, and the difference in the number of communicators went away.

Two lessons I learned.  1. Never make assumptions about how the network communication "should" be implemented.  2. A dedicated engineering driving testing can uncover a lot of interesting issues.

No comments:

Post a Comment