Moving things around

| 1 Comment

This past weekend I moved our datacenter about 600 feet to a new building. The last time I participated in a similar move was in November 2003, and I was still a junior sysadmin. This time I was the main organizer behind it, and there are a lot of moving parts. Unlike the last move, we weren't building a new location, we were just moving to a different facility.

That still requires moving a bunch of stuff. As it happens, in terms of device count this move may have been a bit larger than the other one, even though it was a MUCH larger datacenter moving. Things have miniaturized a lot in the last 9ish years since that last move. 74 serial-number carrying devices transferred in this one, that much I know.

Overall the plan survived contact with reality. The bits that tried to throw us off:

  • The packing phase took about 2.5 longer than planned.
  • The network reconfiguration failed due to incompatible cables (the same cables were on both device's spec-list, you'd think they'd work together)
  • The blade-rack really, really didn't  want to come along and had to be convinced with a rubber mallet (it survived). That cost us a lot of time.
  • We ran out of label tape late Saturday

For a move of this magnitude things actually went pretty well, it just took longer than projected. Even though we did have some problems, we were able to improve a few things along the way:

  • I bought shorter power-cables to connect systems together. Those standard 6-foot/2-meter cables are too long for pretty much everything we use. I could use a lot more of them, but the ones I had were able to make one rack really neat.
  • Rearrange some of the network stack for better cabling.
  • Make room for more growth in the rack holding the storage array
  • More redundant power options.
  • Better power-budget overall!

I hope I can go another 9 years before doing this again, but now that I've done it once, it should go easier the next time.

1 Comment

The last time I participated in a similar move was in November 2013, and I was still a junior sysadmin.

Woah... I can't wait to hear about that move... It must have been FAST!

But seriously, I've been involved in many data center moves. One of them very similar. One building to another. It took a huge amount of time compared to what we expected. And because it was a brand new network, there was an inordinate number of issues just getting everything working right.

I've learned that no matter how much time you think you are going to need, at the end of the day, you will be wrong.

The only data center moves that I have seen go reasonable smoothly... are the ones that are simple DNS or BGP changes. You know, those that you have all new hardware that was fully configured and tested prior to the move. Yup, those go great! Unless someone forgot to update the TTL in DNS ahead of time...