Worst-case thinking is something that Sysadmins are kind of prone to. We all know what level of disaster would cause us to lose everything, and it's not a good feeling. At my last job I was asked once what my worst-case scenario was. And it was a truck-bomb in the wrong spot that would cause our datacenter to suddenly drop a few floors, as well as do serious damage to most of our offices (and note, this was asked AFTER 9/11).
Fixing that was easy, don't allow traffic on that road. But that wasn't an option for us. So we just lived with it.
Having been around enough people worrying about this, the thinking goes that if we mitigate the worst-case we also mitigate the bad-cases too. Let's take a look at this, shall we?
If we HAD been able to stop traffic on that road, it would have done nothing for certain other just as costly incidents. A direct hit by a tornado would render the building structurally uncertain for a week or two as the engineers assessed its soundness, and that would cost us quite a lot thank you. A sprinkler release on the floor above the datacenter could cause water to fall into the datacenter, which would be bad. A fire on the same floor as the DC would cause a sprinkler release in the datacenter (no FM-200 system there!) and short a bunch of stuff out. None of this would have been mitigated by stopping traffic on that one road.
WWU is the kind of enterprise where physical presence is required for most of our business. The kind of disaster that would limit our ability to teach while not also affecting our classrooms themselves limits the kind of disaster to plan for. As it happens, cutting two fiber runs would stop most network-based instruction, so that's the disaster we plan for. This building sinking into the bog it was built on is... a dark fantasy, and only likely in the kind of earthquake that'd also do serious damage to campus itself.
So yes. Good risk-management involves looking at the probable risks, not the worst-case risks and hoping good overall coverage inherits from that.
Fixing that was easy, don't allow traffic on that road. But that wasn't an option for us. So we just lived with it.
Having been around enough people worrying about this, the thinking goes that if we mitigate the worst-case we also mitigate the bad-cases too. Let's take a look at this, shall we?
If we HAD been able to stop traffic on that road, it would have done nothing for certain other just as costly incidents. A direct hit by a tornado would render the building structurally uncertain for a week or two as the engineers assessed its soundness, and that would cost us quite a lot thank you. A sprinkler release on the floor above the datacenter could cause water to fall into the datacenter, which would be bad. A fire on the same floor as the DC would cause a sprinkler release in the datacenter (no FM-200 system there!) and short a bunch of stuff out. None of this would have been mitigated by stopping traffic on that one road.
WWU is the kind of enterprise where physical presence is required for most of our business. The kind of disaster that would limit our ability to teach while not also affecting our classrooms themselves limits the kind of disaster to plan for. As it happens, cutting two fiber runs would stop most network-based instruction, so that's the disaster we plan for. This building sinking into the bog it was built on is... a dark fantasy, and only likely in the kind of earthquake that'd also do serious damage to campus itself.
So yes. Good risk-management involves looking at the probable risks, not the worst-case risks and hoping good overall coverage inherits from that.