February 2013 Archives

What I did this last Tuesday

Last Tuesday was a Grand Unified Meetup of the DC-area DevOps-like groups. Lopsa-DC was there, as was Crabby Admins, DevOpsDC and a bunch of others I hadn't heard of before. It was great. And now the YouTube videos are posted!

The first presentation on Ops School:

Ops School is something I first heard about at this last LISA conference, and even spoke with the guys for a while. This is a project that sprang out of Etsy, but is worth it for the rest of us. While there are some sysadminly degree programs out there, the vasty majority of us learned it on-the-job as it were. Ops School is aiming to build an actual curriculum for people to try and get basically fluent or to brush up on areas we're not all that familiar with.

I already have ideas on how I can contribute.

The second talk, on how to have a career in what it is that we do:

Occasionally profane, but a good overview of how we have jobs in this thing. And a critique of the whole DevOps idea in the first place, which is a very nice contrast to the gestalt view I've picked up elsewhere.

DevOps is all about teaching Ops to Dev.

Theo's point is that Ops is something that everyone needs to get fluent in. That's how a business, especially a web-oriented one, survives. It means that Dev learns to be cautious about making changes to live systems. It means the sysadmins learn that certain kinds of outage (the partial, some-features-are-disabled kind) can be more damaging to a company than a full-stop outage.

And Ops is not the sole purview of the sysadmin-staff, the Sales staff are extremely ops-oriented. It means customer care is a major factor in deciding which changes are worth the risk, and it's usually not the Sysadmin staff who know best what the customers want.

Anyway, watch the video.

Moving things around

| 1 Comment

This past weekend I moved our datacenter about 600 feet to a new building. The last time I participated in a similar move was in November 2003, and I was still a junior sysadmin. This time I was the main organizer behind it, and there are a lot of moving parts. Unlike the last move, we weren't building a new location, we were just moving to a different facility.

That still requires moving a bunch of stuff. As it happens, in terms of device count this move may have been a bit larger than the other one, even though it was a MUCH larger datacenter moving. Things have miniaturized a lot in the last 9ish years since that last move. 74 serial-number carrying devices transferred in this one, that much I know.

Overall the plan survived contact with reality. The bits that tried to throw us off:

  • The packing phase took about 2.5 longer than planned.
  • The network reconfiguration failed due to incompatible cables (the same cables were on both device's spec-list, you'd think they'd work together)
  • The blade-rack really, really didn't  want to come along and had to be convinced with a rubber mallet (it survived). That cost us a lot of time.
  • We ran out of label tape late Saturday

For a move of this magnitude things actually went pretty well, it just took longer than projected. Even though we did have some problems, we were able to improve a few things along the way:

  • I bought shorter power-cables to connect systems together. Those standard 6-foot/2-meter cables are too long for pretty much everything we use. I could use a lot more of them, but the ones I had were able to make one rack really neat.
  • Rearrange some of the network stack for better cabling.
  • Make room for more growth in the rack holding the storage array
  • More redundant power options.
  • Better power-budget overall!

I hope I can go another 9 years before doing this again, but now that I've done it once, it should go easier the next time.

And not just kids

This commentary on the role of technology in the educational market....

[Stone Soup 11/4/2012 -- used with permission]

....is just as appropriate for knowledge-workers on sick-days.

Your nose may be flowing freely, your voice gone w-a-y past sexy base-voice and into sepulchral, and you really shouldn't be more than 20 feet away from a bathroom... but that's no excuse not to get a couple of productive hours out of the way from the comfort of your home. No whittling down the Netflix queue for you, slacker.

There is some good and bad here. Some illnesses are merely unpleasant but highly contagious, so staying out of the office (and in my case, public transit as well) is more of a public health thing than a "rest will get you back to work faster" thing. Some make the act of getting to work impossible (20 ft bathroom restriction, for example) but don't impact ability to think. For these kinds of illness, being productive on a 'sick' day is entirely possible.

The bad side comes when a workaholic who really should be spending 16 hours sleeping actually spends 9 hours in front of the computer. Exhaustion will make their work product crap, and they'll likely be sick longer due to lack of recuperating sleep. These are the kinds of people that managers really should say, "...and I don't want to see you online," to. I have seen this exact thing happen, and it turned a full-on working-from-home person into someone who only replied to email twice in a day. It works, so those of you with direct-reports, keep this in mind.

But the thing is... it's so easy to let this kind of workaholic just work. They're an adult, goes the thinking, they can make their own decisions. In the era before telecommuting was common, managers had to forcibly evict these people lest they contaminate the office and cause half the office to be out sick a week later. There is even a term for this: Presenteeism.

Sometimes you really do have to tell someone, "It's OK to take a nap, really. We'll both be better for it."