The dragon in the datacenter

Systems Administrators have a reputation, a bad one, when it comes to personal skills. I saw it at WWU when problems went unreported because users were afraid we'd yell at them for being stupid. I see it every time someone speaks with passion about DevOps improving the adversarial relationship between Dev and Ops. Two different groups of people, two different problems, same root cause.

  1. Not formally trained people experiencing problems we're tasked with fixing (a.k.a. "users").
  2. Formally trained engineers trying to build/maintain a complex system (a.k.a. "dev").

Dealing with the untrained

End users are tricky people. They don't think the way we do. Because they don't know how a system works, they develop completely wrong mythologies for why things break the way they do. They share folk remedies with each other rather than calling for trained assistance. Those folk remedies can actually make things worse.

Dealing with the trained

Developers are tricky people. They're supposed to understand this stuff, but for some reason only get part of it. Or they only really see one part of the whole constellation of the problem-space and don't understand how their actions make things difficult for another part of the puzzle. It's forever frustrating because they're supposed to know better.


Cynicism: (1): The firm belief that the person telling you how to do something differently is blowing smoke up your ass because they don't know it doesn't work that way.
(2): The firm belief that a certain class of person will just never, ever, get it.


Sysadmins become jaded cynics because the end users never get any better, and explaining the same thing over and over again gets old. And it never helps. And they keep doing the same stupid things, over, and over, and over. No amount of training helps. No amount of "intuitive" walk-throughs help. No amount of video tours help. The customer support organization helps filter the blithering lunacy, but it just means the extra special stupid escalates to L3 where we live.

Customer Service is an outlook as much as it is a skill. Far too many of us lack that outlook and aren't motivated to get the skill. The 'customer' we're serving most of the time is an abstract known as "uptime", that's quantifiable and doesn't file reports with your boss when you get a bit firm with it over the phone. As an industry we're regular consumers of Customer Support in the form of our vendors and the support contracts we hold with them. We know what we like when we get to the human:

  • They speak our language.
  • They don't get defensive when we blow steam about our frustrations with their product.
  • When we describe in detail what we think the problem is they don't dismiss our concerns and tell us how it really failed.

The jaded cynic sysadmin doesn't do any of that. We use condescending language (very probably unintentionally condescending). We respond to attacks on our systems by getting defensive. We see a chance to myth-bust and jump on it with glee, describing in detail how that failure mode actually occurred.

When users have problems they don't come to the jaded cynic sysadmin with them. This is driven through a combination of fear of being attacked, disgust that such people are allowed to keep working, and a desire to avoid assholes whenever possible.


Corrosive Cynicism: The belief that everyone around you doesn't know how it really works, and it's your job to explain why that is.


Sysadmins become jaded cynics after the developers persistently and stubbornly refuse to pick up the little platform quirks they're developing the application on. It gets tiring having to continually disabuse them of their assumptions on how the OS/platform works. You wish they'd talk to you sooner rather than wait to the end when all the bad assumptions have been baked in and they have to patch around them.

This is not some ever-changing population of end users, these are your coworkers. You see them every day (or, well, at least once or twice a week at meetings). You're both supporting the same overall problem, but your focus areas are different. They're concerned with algorithmic efficiency, you're concerned with system resources and what consumption rates mean for the future. They're concerned with making this one application work, you're concerned with how that application will fit in to the whole ecosystem of apps that share the same resources.

No one understands how it all fits together but you and your fellow sysadmins. If they came to you earlier, they wouldn't have these problems.

Congratulation, you're a BOFH.

The failure-mode here is the same as it was with the end users, a lack of Customer Service skills. Only instead of an ever changing population of stupid-doers you have a small population of the willfully ignorant. If you become hard to approach, you'll be fixing messes well after it was cheap and easy to fix. They're avoiding you because you're forever telling them 'no', and you're not exactly nice about it.


From the point of view of others

Green Dragon
Alignment Lawful Evil
Breath Weapon Acid Cone
Preferred Habitat Forests and Datacenters

The jaded cynic sysadmin most definitely works within the system. They may even be the system, but that authoritiy is derived from someone who let them have the keys to the kingdom. However, they're very often the last word when it comes to their systems This makes them lawful.

The jaded cynic sysadmin never seems to care what others think. They have their own goals, and asking them for stuff doesn't seem to do anything. Bribery can work, though. This makes them evil.

The jaded cynic sysadmin is... not someone you want to piss off. And they're easy to piss off, just existing seems to be enough sometimes. When that happens you risk a verbal flaying. It's called a breath weapon for a reason.