SysAdmins have no trouble making big lists of what can go wrong and what we're doing to stave that off a little longer. The tricky problem is pushing large organizations to take a harder look at systemic risks and taking them seriously. I mean, the big companies have to have disaster recovery (DR) plans for compliance reasons; but there are a lot of differences between box-ticking DR plans and comprehensive DR plans.
Any company big enough to get past the running out of money is the biggest disaster phase has probably spent some time thinking about what to do if things go wrong. But how do you, the engineer in the room, get the deciders to think about disasters in productive ways?
The really big disasters are obvious:
- The datacenter catches fire after a hurricane
- The Region goes dark due to a major earthquake
- Pandemic flu means 60% of the office is offline at the same time
- An engineer or automation accidentally:
- Drops all the tables in the database
- Deletes all the objects out of the object store
- Destroys all the clusters/servlets/pods
- Deconfigures the VPN
- The above happens and you find your backups haven't worked in months
All obvious stuff, and building to deal with them will let you tick the box for compliance DR. Cool.
But there are other disasters, the sneaky ones that make you think and take a hard look at process and procedures in a way that the "oops we lost everything of [x] type" disasters generally don't.
- An attacker subverts your laptop management software (JAMF, InTune, etc) and pushes a cryptolocker to all employee laptops
- 30% of your application secrets got exposed through a server side request forgery (SSRF) attack
- Nefarious personages get access to your continuous integration environment and inject trojans into your dependency chains
- A key third party, such as your payment processor, gets ransomwared and goes offline for three weeks
- A Slack/Teams bot got subverted and has been feeding internal data to unauthorized third parties for months
The above are all kinda "security" disasters, and that's my point. SysAdmins sometimes think of these, but even we are guilty of not having the right mental models to rattle these off the top of our head when asked. Asking about disasters like this list should start conversations that generally don't happen. Or you get the bad case: people shrug and say "that's Security's problem, not ours," which is a sign you have a toxic reliability culture.
Security-type disasters have a phase that merely technical disasters lack: how do we restore trust in production systems? In technical disasters, you can start recovery as soon as you've detected the disaster. For security disasters recovery has to wait until the attacker has been evicted, which can take a while. This security delay means key recovery concepts like Recovery Time and Recovery Point Objectives (RTO/RPO) will be subtly different.
If you're trying to knock loose some ossified DR thinking, these security type disasters can crack open new opportunities to make your job safer.