Today I have a very special interview. I have with me Steve, who is
responsible for North Pole IT operations. As you can imagine, he is
rather busy this time of year so I'm glad he had a chance to talk
with me.
sysadmin1138: Thanks again for talking with me. But first, the question on all of our minds, how in the heck did you LAND that job?
Steve: I just sort of fell into it. I was on the Admunsen-Scott over-winter team for 2007 and 2008. You know, the South Pole station? It was the day after the Winter Solstice in 2008, which is June 22nd for you northern-hemisphere people, recovering from a massive hangover from the party the night before when a Christmas Elf showed up in my room.
sysadmin1138: That must have been a big shock.
Steve: You have no idea. He was about four feet tall, big, pointy ears, green felt hat and jacket, with yellow pants. You'd scream too if you saw that staring at you when you woke up. The elf assured me I was quite sober, a fact I was painfully aware of at that point. He passed me a mug of egg-nog and told me what he was doing there. The Big Man liked what I was doing down here, was looking to modernize his operation some, and would I be willing to be part of it? What with me already familiar with arctic IT and all.
Turns out that nog was spiked, which helped with the hangover a bit, and helped settle my nerves. Yes, they really were looking for a systems engineer. They already had an IT plant of sorts but the old guy had expressed a desire to retire and see more of his grandkids. This was just too loony for reality so I said, "Pfft, sure." He companionably clapped me on the back, shook my hand, and left.
It was only the next day when the nog-mug was still there when I got up that I got the inkling that perhaps, maybe, that elf was actually in my room and not just in a room in my head.
It turns out that mug shouldn't have gone in the dish-washer. That was the period to my suspicion that I'd dreamed the whole thing. I mean really, who dreams up breaking a mug in a dish-washer?
sysadmin1138: I've dreamed up UPS alarms, but not mugs-in-the-dishwasher I have to admit.
sysadmin1138: Thanks again for talking with me. But first, the question on all of our minds, how in the heck did you LAND that job?
Steve: I just sort of fell into it. I was on the Admunsen-Scott over-winter team for 2007 and 2008. You know, the South Pole station? It was the day after the Winter Solstice in 2008, which is June 22nd for you northern-hemisphere people, recovering from a massive hangover from the party the night before when a Christmas Elf showed up in my room.
sysadmin1138: That must have been a big shock.
Steve: You have no idea. He was about four feet tall, big, pointy ears, green felt hat and jacket, with yellow pants. You'd scream too if you saw that staring at you when you woke up. The elf assured me I was quite sober, a fact I was painfully aware of at that point. He passed me a mug of egg-nog and told me what he was doing there. The Big Man liked what I was doing down here, was looking to modernize his operation some, and would I be willing to be part of it? What with me already familiar with arctic IT and all.
Turns out that nog was spiked, which helped with the hangover a bit, and helped settle my nerves. Yes, they really were looking for a systems engineer. They already had an IT plant of sorts but the old guy had expressed a desire to retire and see more of his grandkids. This was just too loony for reality so I said, "Pfft, sure." He companionably clapped me on the back, shook my hand, and left.
It was only the next day when the nog-mug was still there when I got up that I got the inkling that perhaps, maybe, that elf was actually in my room and not just in a room in my head.
It turns out that mug shouldn't have gone in the dish-washer. That was the period to my suspicion that I'd dreamed the whole thing. I mean really, who dreams up breaking a mug in a dish-washer?
sysadmin1138: I've dreamed up UPS alarms, but not mugs-in-the-dishwasher I have to admit.
Steve: See? I know! That's when I started getting worried. What was going to happen? On crew change-out day, was Santa going to land his sleigh next to the [Ski-landing equipped LC-130], say, "Ho Ho Ho! You're coming with me Steve! Hop right in!" and we'd fly out?
It ended up being downright boring. Two weeks before change-out, there was an old-fashioned envelope on my dresser. Nice old paper, hand caligraphy on the front with my name on it and 'employment contract' in smaller letters. I opened the envelope and there it was. Ten pages, double-sided. Old-school thick paper, and the best part? Ten pages of hand written legal contract. With line numbers. And sign-here flags. And let me tell you, the penalty clauses on his NDA were the worst I'd ever seen.
sysadmin1138: Anything you can share, or is that also under NDA?
Steve: Well, let's just say you get put on the "Very Naughty" list, and don't come off of it. Ever.
sysadmin1138: Oo, that doesn't sound pleasent.
Steve: No, it isn't.
As I was saying, the contract was on the up and up, and pay was.... worth the risk. Also, no longer a government contractor so that was kinda nice. So I signed. How it'd work was I'd fly out, land at the staging area in Argentina, and head to a certain field at midnight. Which I did. I was met by a pair of Elves, who escorted me to where the sleigh was waiting. An hour later, I was at ninety degrees north.
sysadmin1138: That's very fast.
Steve: Well, those reindeer kinda have to be.
sysadmin1138: How does the North Pole differ from the South Pole?
Steve: Pfft, well for one you're at sea-level not twelve thousand feet, that makes a big difference. Second, the building foundations were in rock not ice, which makes things a lot more stable.
sysadmin1138: That's surprising. Those pylons have to go hundreds of feet deep and deal with shifting sea-ice. That's some nice engineering.
Steve: Hah! Let me ask you. Remember that Christmas TV special about Rudolph? What can you tell me about the geography?
sysadmin1138: Lots of pointy ice-mountains, evergreen trees, and.... rocky mountains.
Steve: Yep. That's a better description of Northern Canada than the actual North Pole. But... that's what Santa's North Pole looks like. It's at ninety degrees north, just a bit to the left. Magic!
Also, it's a lot warmer. Winter temps rarely go below -40 C, which is warmer than the actual Pole's weather, and summer temps sometimes kiss freezing.
I've yet to find Superman's Fortress of Solitude, but I keep looking.
sysadmin1138: Tell us a bit about the IT challenges you face up there. What kinds of workloads are you managing?
Steve: The biggest one is Naughty/Nice list updates. With, like, six billion people on the planet now that's a lot of updates we have to handle in a day. Very highly transactional and with a hard delivery date. The last fully manual list was run in 1974, and it went fully electronic in 1980.
Fun story there. The Elves unionized in 1980 to protest working conditions. There just weren't enough hours in a day to get all the updates done that needed to be done, and also keep up with toy quota. There were a few work-stoppages during that battle. Which meant that for Christmas 1980, Santa went out with an insufficiently updated list, and a lot more toys from Kenner than anyone was comfortable with. I still hear stories about that year, even though I still treasure the X-Wing I got then.
The second biggest is simple inventory control. Request, fulfullment, the whole nine yards. Amazon does more volume than us, so look to them for how they solve it. You'll see similar methods.
sysadmin1138: So, distributed warehouses and trackable shipments?
Steve: I don't want to go onto the Very Naughty list, so, er, no comment.
sysadmin1138: Point taken. Seeing that you're a little to the left of the North Pole, how do you keep your infrastructure powered?
Steve: They've had a geo-thermal tap for a long time now. Before then, they used, get this, the reindeer.
sysadmin1138: The reindeer??
Steve: Look at it this way. Those reindeer have to go HOW far in a single day? They have to stay in shape some how, so why not hitch them up to a treadmill and get some power out of them? Before they electrified they used the same kind of thing to drive the assembly-line belts.
sysadmin1138: The only unionized sysadmins I've ever known were civil-service. Are you in the Elf Local?
Steve: No... not by a long shot. My august predicessor was hired to drive the IT stack outside the Union framework. And that's where I've stayed ever since.
sysadmin1138: What kind of internet connection do you get up there?
Steve: Actually, we don't HAVE one. We're air-gapped. This makes my life harder every year as more and more software, and system software, assumes an Internet connection. On the one hand, hackers and crackers and thieves oh my can't mess with my shit. On the other hand, any time I need OS updates I need to step into the real world to download them.
Hell, when I got there we were still running a UUCP relay! Turns out there is a lot of sense to that, but modern SMTP systems can't handle those old bang-paths anymore so I had to do something else.
sysadmin1138: So how are you handling email?
Steve: Simple. SMTP is a store-and-forward system. I have a real world server that downloads and queues our incoming mail. This mail server doesn't send a DSN after four hours, it only complains after a day. We then forward our queued outgoing mail through it. We're shuttling a, er, laptop between locations. It works. Reindeer express!
sysadmin1138: How do you handle equipment replacemnt?
Steve: Getting into 'Very Naughty' territory here. But suffice it to say, I ask for something, if it gets approved, it'll show up. I don't get everything I ask for, I do need to justify it.
sysadmin1138: How about parts replaceent? When something breaks I imagine you're kinda far from a supply depot.
Steve: Had this problem at [the South Pole]. We maintain a parts warehouse for all components. We don't go through disk-drives like we do down there, but we still go through a lot. Every so often we obtain parts refreshes. Everything has a spare!
sysadmin1138: What does your IT department look like? I can't imagine you're all by your self up there.
Steve: You're not wrong. There are two other non-Elves working with me. One is a DBA who they hired in 2001. He doesn't talk about where he came from, but his job is writing the database routines that handle both The List and the inventory system. And, thankfully, all of their licensing. I just keep him in hardware and operating systems and get out of his way. He specifically told me not to reveal his name though, so take that as you will. He scares me.
The other one is Karen, who is our systems programmer. She's been here since 1991 and writes most of the Elf interface stuff. She phased out the old Cobol stuff when she got here and has been in the land of C and C++ ever since.
Plus a bunch of elves.
Fun fact: I use half-height racks exclusively. When your workforce tops out at 48 inches, anything above rack unit 30 isn't usable without a ladder.
sysadmin1138: How does working with elves change things?
Steve: They're just people, but with mad hand-crafting skills. Take my racks...
What are elves? Workmen. For centuries they built toys. Lots of them. Fast. And high quality. Take that and apply it to a wiring problem.
What you get is beauty. Absolute, unalloyed beauty. Every cable, power or network, cut exactly to the right length and perfectly terminated. Even fiber optics. Even, I add, knowing you'll get the magnitude of this, CX4 ends. They work every time. Perfectly. No spooled cables where there shouldn't be any. Custom built strain reliefs to ensure cables never exceed their bend-radius or experience pinching. I wish I could show you pictures of my racks because oh my god.
Also? They are consumate workaholics and only sleep four hours a night. I don't even try to keep up with them, I just can't.
Makes you think about how hard things were back in '79 when they formed the union.
sysadmin1138: Now I really want to see those pictures.
Steve: I know! I haven't done wiring since Admunsen-Scott, no point in me mucking up that artistry. I am now completely and totally spoiled for wire-management.
sysadmin1138: On the one hand, you've got a lot more people to work with than you did at the South Pole. But you have less internet access. How does that impact any sense of isolation you have?
Steve: I work with a lot of great people, and we all get along well. But the lack of internet access is becoming an issue. There are a bunch of things I know I'm missing out on and I wish I could do. Things like Twitter. Even Astronauts get better access than we do. Email is one thing, but being connected is soon going to be an operational need. I have some ideas for that, but none of them are easy.
sysadmin1138: What are your busy times of year?
Steve: For me? January, that's when I get to do all the big upgrades. The big run is done, the warehouses are being cleaned, and the elves take a few weeks off. The List is still updated, but we tolerate more queueing during this period.
December I spend on fire-watch. Which is why I have time to talk to you. Everyone else is under mad pressure right now to meet deadline.
Another fun fact: June 25th we have a small run to the Southern Hermisphere. A bunch of people celebrate "the Santa holiday" during southern winter.
sysadmin1138: Any closing remarks?
Steve: Santa really does care for people. Like, a lot. The whole labor thing was to meet rising demand for his services and he got a bit too selfless and had to be pulled back by his workforce. It hurts him when people do naughty things. Really. Be nice!
Thanks for listening to me rant.
sysadmin1138: Any time.