You may have noticed a few glitches accessing core services in the last week. Well, we've figured out where they are occuring, but not why. They're backup related, and they're also CatalystOS related. I'm not the telecom geek, so I'm not 100% on the details relating to the switch/routing infrastructure. But this is the diagram of our traffic problems.
ServerA is being backed up by BACKUP. The traffic follows this path:
Switch 1 -100-> PIX -100-> Switch 2 -Gig-> Switch 3 -Gig-> Router 1 -Gig-> Switch 2 -Gig-> Switch 4
Annotated...
Switch 1 -> PIX Getting out of the firewalled network
Pix -> Switch 2 Because that's where the PIX is plugged in.
Switch 2 -> Switch 3 Switch 2 sees that the traffic is from VLAN 1, so it sends it to the switch that handles VLAN 1 traffic.
Switch 3 -> Router 1 Switch 3 sees that the traffic is destined for VLAN 2, so it shoves it at a router
Router 1 -> Switch 2 Router sees that the traffic is destined for VLAN2 and knows that Switch 2 handles that stuff, so sends it on to Switch 2
Switch 2 -> Switch 4 Switch 2 knows that Switch 4 is hosting the port the traffic needs to get to, so shoves it down the ISL.
How the traffic SHOULD route is like this...
Switch 1 -> PIX -> Switch 2 -> Switch 4
The problem is with switch 2. The traffic is not enough to saturate it, even with handling the data stream twice. The CPU is not bombing. Somehow, when it is doing this backup procedure it stops processing all other traffic but the backup traffic. We're trying to figure out why. This is strongly corolated with the CatOS upgrade we did last week. CatOS upgrades are major mojo, so we're not about to backrev just for this if there are workarounds we can use instead.
ServerA is being backed up by BACKUP. The traffic follows this path:
Switch 1 -100-> PIX -100-> Switch 2 -Gig-> Switch 3 -Gig-> Router 1 -Gig-> Switch 2 -Gig-> Switch 4
Annotated...
Switch 1 -> PIX Getting out of the firewalled network
Pix -> Switch 2 Because that's where the PIX is plugged in.
Switch 2 -> Switch 3 Switch 2 sees that the traffic is from VLAN 1, so it sends it to the switch that handles VLAN 1 traffic.
Switch 3 -> Router 1 Switch 3 sees that the traffic is destined for VLAN 2, so it shoves it at a router
Router 1 -> Switch 2 Router sees that the traffic is destined for VLAN2 and knows that Switch 2 handles that stuff, so sends it on to Switch 2
Switch 2 -> Switch 4 Switch 2 knows that Switch 4 is hosting the port the traffic needs to get to, so shoves it down the ISL.
How the traffic SHOULD route is like this...
Switch 1 -> PIX -> Switch 2 -> Switch 4
The problem is with switch 2. The traffic is not enough to saturate it, even with handling the data stream twice. The CPU is not bombing. Somehow, when it is doing this backup procedure it stops processing all other traffic but the backup traffic. We're trying to figure out why. This is strongly corolated with the CatOS upgrade we did last week. CatOS upgrades are major mojo, so we're not about to backrev just for this if there are workarounds we can use instead.