On living as Root

Yesterday on Slashdot, one of the Ask Slashdot questions was: " In your experience, do IT administrators abuse their supervisory powers?"

That's a good question. BOFH humor aside, it has been my experience that the large majority of us don't do so intentionally. Most of what happens is the petty stuff that even regular helpdesk staff do, like take home enterprise license keys. We shouldn't do that, and licensing technology is improving to the point where such pilferage is becoming a lot easier to detect; at some point Microsoft will blacklist some large org's enterprise key for having been pirated and woe unto the IT department that lets that happen.

But what about IT administrators?

First, IT Administrators come in many types. But I'll focus on my own experiences living with enhanced privs. As it happens, I've spent the large majority of my IT career with a user account with better than average privs.

File Access

I can see everything! One of the harder things to keep in mind is what files I can see as me, and what files I can see as my role as sysadmin. This can be hard, especially when I'm rooting about for curiosity. We still add my user to groups even though I can see everything, and I consciously limit myself to only those directories those groups have access to when privately data-mining. You want this. This is one of the top hardest things for a new sysadmin to get used to.

With my rights it is very easy for me to pry into HIPPA protected documents, confidential HR documents, labor-relations negotiations documents, and all sorts of data. I don't go there unless directed as part of the normal execution of my duties. Such as setting access controls, troubleshooting inaccessible files, and restoring data.

I haven't met any sysadmins who routinely spelunk into areas they're not supposed to. They are out there, sadly enough, but it isn't a majority by any stretch.

Email Access

I read your email, but only as part of my duties. Back when we were deploying the Symantec Brightmail anti-spam appliances I read a lot of mail tagged as 'suspect'. I mean, a lot of it. It took a while to tune the settings. Even just subject-lines can be damning. For instance, the regular mails from Victoria Secret were getting flagged as 'suspect' so anyone who ordered from them and used their work account as the email account was visible to me. A BOFH would look for the male names, print out the emails, and post them on the office bulletin board for general mockery. Me? I successfully forgot who got what.

One gray area is the 'plain view' problem. If I'm asked to set or troubleshoot Outlook delegates on a specific mailbox, I have to open their mailbox. During that time certain emails are in plain view as I navigate to the menu options I need to go to in order to deal with delegates. Some of those emails can be embarrassing, or downright damning. So far I don't officially notice those mails. Very happily, I've yet to run into anything outright illegal.

Another area that has me looking for specific emails is phishing. If we identify a phishing campaign, the Exchange logs are very good at identifying people who respond to it. I then take than list and look for specific emails in specific mailboxes to see what exactly the response was. While this also has the plain-view problem described above, it does allow us to identify people who gave legitimate password info, and those replying with derision and scorn (a blessed majority). Those that reply with legitimate login info get Noticed.

Internet Monitoring

This varies a LOT from organization to organization. WWU doesn't restrict internet access except in a few cases (no outbound SMTP, no outbound SMB), so we're not a good example. My old job was putting into place internet blockers and an explicit login before access to the internet was granted, which allowed very detailed logs to be kept on who went where. As it happened, IT was not the most privileged group; that honor was held by the Attorney's office.

While IT was restricted, I knew the firewall guys. They worked two cubes down. So if I needed to access something blocked, I could walk down the hall and talk to them. I'd have to provide justification, of course, but it'd generally get granted. The fact that I was one of the people involved with Information Security and helped them make the filters unavoidable helped in this.

But the Slashdot questioner does make a good point. Such IT sites do generally get let through the filters. I strongly suspect this is because the IT users are very close to the managers setting filtering policy so are able to make the convincing, "but these sites are very useful in support of my job," arguments. Sites such as serverfault and stackoverflow are very useful for solving problems without expensive vendor contracts. Sites supporting the function of non-IT departments are not so lucky.

Whether or not the grand high IT Admins get unrestricted access to the internet depends a LOT on the organization in question. My old place was good about that.

Firewall Exceptions

This is much more of a .EDU thing since we're vastly more likely to have a routable IPv4 address on our workstation than your non-educational employers. In smaller organizations where your server guys are the same guys who run the network, good-ole-boys comes into play exceptions are much more common. For larger orgs like ours that have server-admin and network-admin split out, it depends on how buddy-buddy the two are.

This is one area where privilege hath its perks.

As it happens, I do have the SSH port on my workstation accessible from the internet. The firewall guys let me have that exception because I also defend servers with that exception, and therefore I know what I'm doing. Also, it allows me into our network in case either VPN server is on the fritz. And considering that I manage one of the VPN servers, having a back-door in is useful.

Other areas

Until a couple weeks ago the MyWeb service this blog used to be served from was managed by me. Which meant I got to monitor the log files for obvious signs of abuse. Generally, if something didn't break the top 10 access files I officially didn't notice. If a specific file broke 25% of total traffic, I had to take notice. Sometimes those files were obviously fine files (home shot video pre-YouTube), others (MP3 archives, DIVX movies) were not so innocent.

One day the user in question was a fellow IT admin. This was also the first time I saw staff doing this, so the protocols were non-existent. What I did was print off the report in question, circle the #1 spot they occupied, and wrote a note that said, in brief:
If this had been a student, the Provost would have been notified and their accounts suspended. The next time I'll have to officially notice.
And then put it on their chair. It never happened again.

Another area is enterprise software licenses. I mentioned that at the top of this post, but as more and more software gets repackaged for point-n-click distribution fewer and fewer IT staff need to know these impossible to memorize numbers. Also helping this trend is the move towards online License Servers, where packages (I'm thinking SPSS right now) need to be told what the license server is before they'll function; you can't take something like that home with you.

Things like Vista or Windows 7 activation codes are another story, but Microsoft has better tracking than they did in the XP days. If you activate our code on your home network, Microsoft will notice. The point at which they'll take action is not known, but when it does all Vista or Windows 7 machines we have will start throwing the, "your activation code has been reported as stolen, please buy a new one," message.

The software industry as a whole is reconfiguring away from offline activation keys, so this 'perk' of IT work is already going by the way-side. Yes, taking them home has always been illegal in the absence of a specific agreement to allow that. And yet, many offices had an unofficial way of not noticing that IT staff were re-using their MS Office codes at home.

That got long. But then, there are a lot of facets to professional ethics and privilege in the IT Admin space.


promoting open source software would solve most of those problems with corporate licenses ...

Over the years I've developed a pretty good knack for forgetting what I've seen and read. I think it's a skill that really needs to come with the job, the deliberate and conscious separation of what is required by the role from what is required as an individual.At an educational establishment I worked for e-mail was described in the AUP as a "postcard system", along with the phrase that we'd never deliberately open e-mails except where absolutely essential for the running of the service. In theory with a post card it's only ever read by the intended recipient but there is always the chance that those involved in delivering might deliberately or inadvertently read the content.I can remember only one specific instance where I deliberately abused my privileges.I'd applied for an internal job and it was well known amongst the staff that HR had a habit of 'forgetting' to pass on resumes from internal staff to managers, and then claiming to affected staff that they'd never received it.I very deliberately logged in to the mail server, su'd up and grabbed relevant mail logs that proved they'd received my e-mail, just in case.Thankfully even though HR had failed to pass on my resume (quel surprise), all it took was contacting the HR director specifically to get things resolved and I didn't have to reveal what I'd done.

A very thoughtful post, and I'm mostly in agreement with you. But I do have to challenge your assertion that Microsoft can somehow detect keys being used by a sysadmin at home. (This being substantially different from a key being widely distributed by said sysadmin.)So, in what way would you expect a Windows 7 installation with my organizations's key on my personal laptop to look different from the very same key on my work issued laptop? If I was using my personal laptop primarily to support the business of the University (which happens a lot with sessional faculty who can't get decent hardware otherwise), then it's not even a violation of the license. There is nothing in the MSCA that say the University has to own the device being licensed.