This weekend I spent 19 hours in the line of duty, doing quite a few
things at our datacenter. We had a 48 hour outage window arranged with
our clients, and some things needed to get done in time for others to
Q/A the changes. Big, big stuff.
For something this large, we needed a checklist to keep track of it all. I spent the previous week honing it into a workable shape. Even though there were two of us available for the work, I still kept it in a serialized form rather than track items that could be done in parallel as that would have complicated things.
And it was a damned good thing I had that list. At hour 14 of the work, I had been 9 hours without food and wasn't tracking things as well as I would had I been fresh. That checklist provided me with brains-on-a-page so I could be smart when I wasn't actually smart. I could focus on important things like why these VLANs weren't working the way I anticipated, and not wrack my brains for what happens next. If we didn't have that checklist, I most definitely would have forgotten key steps.
Checklists. Useful things, especially when you're going to be under stress or are going to be doing complex things.
For something this large, we needed a checklist to keep track of it all. I spent the previous week honing it into a workable shape. Even though there were two of us available for the work, I still kept it in a serialized form rather than track items that could be done in parallel as that would have complicated things.
And it was a damned good thing I had that list. At hour 14 of the work, I had been 9 hours without food and wasn't tracking things as well as I would had I been fresh. That checklist provided me with brains-on-a-page so I could be smart when I wasn't actually smart. I could focus on important things like why these VLANs weren't working the way I anticipated, and not wrack my brains for what happens next. If we didn't have that checklist, I most definitely would have forgotten key steps.
Checklists. Useful things, especially when you're going to be under stress or are going to be doing complex things.
I totally rate checklists. They're useful not just when you're going to be stressed out, but for standard stuff too.
We have electronic checklists built into our asset management system for pretty mundane things like setting up a new PC/laptop. Not only can you do stuff, leave it for a bit and come back, but someone else can also pick it up should you be mowed down by a bus or something.
It's also a bit of a safety blanket for someone like me who doesn't set up too many laptops, but offers a hand if we have a sudden influx of replacements to do. Simple bullet points with links to knowledgebase articles where appropriate mean anyone who is not totally au-fait with the process, but has some degree of IT knowledge can chip in and help.
You should check out the Checklist Manifesto by Dr. Gawande.
Two more really important takeaways from your experience (as they have been from my own) are that (1) you need to take breaks, and (2) you need to eat. It's really easy to get sucked into a problem and keep banging your head on it until you realize you're completely lost or really screw something up, especially when you haven't eaten and your blood sugar is low and your thinking muddled. During an unplanned outage, if you're lucky, you'll have someone who'll check on you periodically and encourage you to take a break if you need one. But during a planned outage, you need to schedule some time for breaks and meals. Put them on your checklist!
And another really good aspect of checklists is that if its done right, it ends up being a nice "as-built" document. I've based all of my deployment planning around this structure, and you kill two birds with one stone. Oh, and its all in my favorite non-technical app; OneNote.
At the risk of a "me too" comment, I totally agree.
Back in the days when I was doing a lot of travel and installing point-of-sale networks at remote sites, I quickly developed a checklist of things to do to make sure we didn't miss anything. I found it especially useful after doing the 10th or 15th install in a row when I was about to shift into "burn out" mode on the whole gig. Week after week of the same basic install really got tedious and checklists helped a lot.
And, as others have mentioned, I've used them as the basis for documentation for the next poor schlub that has to do my job after I move on. I hope they worked well for those guys, too, and that I left the various departments and jobs in better shape than when I got there.
Also? The utility of checklists has been proven by science! Researchers trying to improve treatment in emergency rooms and surgical centers discovered that checklists reduced errors and raised the survivability rate of patients. Not quite the same as our gig, but the results are applicable.