At first, it seemed that every time a machine was broadcasting a UDP datagram on its own local network, the broadcast was being incorrectly picked up by the router and getting forwarded to every other network on the campus. This initially lead us to believe that the Cisco routers that made up the backbone were either misconfigured or buggy (there are certain options on the Cisco that will cause this behavior).
To test this theory, I wrote a program that would send similar UDP broadcasts -- both local broadcasts and directed broadcasts to other networks. First, I wrote udpsend, a simple program that generated UDP datagrams that looked similar to InocuLAN's broadcasts (the same port number and size). Later, when the problem was escalated, I wrote udpsendraw to generate the entire IP datagram from scratch, allowing finer control over the IP header options, in order to more closely imitate the InocuLAN transmissions. Much to our surprise, the packets generated by my program were forwarded correctly, whereas the InocuLAN packets still seemed to get forwarded to every network on campus. We couldn't figure out why the routers were treating my simulated packets differently than the InocuLAN packets, since the packets looked identical.
When we finally isolated one of the offending machines, we saw that each InocuLAN machine was actually sending out multiple directed broadcasts -- one for each subnet on which we had an NT domain! We had not seen this earlier because each of our Ethernets is actually made up of a number of switched segments; since the sniffer I'd been running was not on the same segment as the InocuLAN machines, I had only seen the local broadcasts, not the directed broadcasts destined for other networks.
Thus, the mystery was explained: the routers weren't at fault at all; InocuLAN was just splattercasting its discovery data all over the place. The immediate "fix" (I'm using that term very loosely here) was for us to add router filters at the border of our network to prevent the directed broadcasts from leaving our network. Specifically, we added the following router filters:
access-list 102 deny udp any any eq 41508 log access-list 102 permit ip any anyThis at least prevented us from killing other networks. The next step was to use both my tcpdump-based sniffer and the router's filter logs to find the machines that were generating the broadcasts and reconfiguring Inoculan on those machines to disable autodiscovery. Eventually (months later) we just decided to stop using Inoculan.
While these programs were originally written for a very esoteric purpose, I decided to put them on the Web anyway because they're a good example of how to generate datagrams using the raw socket interface (IPPROTO_RAW).
Back to my software page
Back to my home page
Jeremy Elson