Proxy ARP is enabled on our routers. I'm 100% certain this has saved the bacon of many of the technicians here on campus, since our subnet is in the Class-B range (140.160.0.0/16), and Windows knows this so applies a default subnet of 255.255.0.0 when setting up a static IP address. Without Proxy ARP, a tech who doesn't fix this will soon find that talking to anything on campus doesn't work, but talking to, say, sysadmin1138.net works just fine. With Proxy ARP, it all works just fine and the tech is never the wiser.
We just had this crop up on a server, only with a twist.
It turns out that the F5 BigIP will also issue a Proxy ARP for Virtual Servers that are configured on it. Which means that for some addresses on some subnets, we actually have two network devices issuing Proxy ARP packets. This, as you can well imagine, is sub-optimal. How it works is this, from a Layer 2 point of view...
Mailer: Who has 140.160.243.16? Tell Mailer.
BigIP: 140.160.243.16 is BigIP
Mailer to BigIP: TCP/25 to 140.160.243.16 [SYN]
Cisco: 140.160.243.16 is Cisco
BigIP to Mailer: TCP/43124 to Mailer [SYN/ACK]
Mailer to Cisco: TCP/25 to 140.160.243 [ACK]
Cisco to Mailer: [Reset]
What you're seeing is an ARP update in the middle of the TCP 3-way handshake. The Mailer server dutifully updates its ARP table for 140.160.243.16, which takes it down a different network path than the BigIP expects, and gets a TCP Reset issued.
What was throwing us on this one was that the connection would reset, but subsequent attempts would work just fine. This is because we were still within the ARP timeout value when the second attempt was made, and things just worked, at least for a little while.
Setting the network mask correctly forces the Mailer to realize that 140.160.243.1 is NOT a local address and the traffic transmits correctly through the gateway, and everything works.
We just had this crop up on a server, only with a twist.
It turns out that the F5 BigIP will also issue a Proxy ARP for Virtual Servers that are configured on it. Which means that for some addresses on some subnets, we actually have two network devices issuing Proxy ARP packets. This, as you can well imagine, is sub-optimal. How it works is this, from a Layer 2 point of view...
Mailer: Who has 140.160.243.16? Tell Mailer.
BigIP: 140.160.243.16 is BigIP
Mailer to BigIP: TCP/25 to 140.160.243.16 [SYN]
Cisco: 140.160.243.16 is Cisco
BigIP to Mailer: TCP/43124 to Mailer [SYN/ACK]
Mailer to Cisco: TCP/25 to 140.160.243 [ACK]
Cisco to Mailer: [Reset]
What you're seeing is an ARP update in the middle of the TCP 3-way handshake. The Mailer server dutifully updates its ARP table for 140.160.243.16, which takes it down a different network path than the BigIP expects, and gets a TCP Reset issued.
What was throwing us on this one was that the connection would reset, but subsequent attempts would work just fine. This is because we were still within the ARP timeout value when the second attempt was made, and things just worked, at least for a little while.
Setting the network mask correctly forces the Mailer to realize that 140.160.243.1 is NOT a local address and the traffic transmits correctly through the gateway, and everything works.